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Abstract 


Human-machine  system  performance  can  be  improved  by  using  technologies  that  intelligently 
adapt  the  operator  machine  interface  (OMI)  or  by  task  automation  provided  to  the  operator 
with  external  context  (i.e.,  task  environment)  and  internal  context  (i.e.,  operator  state).  It  is 
challenging  to  design  effective  Intelligent  Adaptive  Systems  (IAIs)  due  to  a  lack  of 
established  design  guidelines.  A  literature  review  was  conducted  to  examine  approaches  to  the 
design  of  IASs,  and  a  framework  was  developed  to  describe  these  design  approaches  using 
consistent  and  unambiguous  terminology.  Combining  methodologies  from  both  Human 
Computer  Interaction  (HCI)  and  Human  Factors  (HF)  fields,  conceptual  and  design 
frameworks  were  developed  to  provide  the  design  and  implementation  of  IASs.  Finally,  a 
number  of  decision  trees  (see  section  12)  were  used  to  select  appropriate  analytical  techniques 
and  design  approaches.  The  proposed  frameworks  provide  guidelines  for  designing  IASs  in 
the  military  domain,  and  broadly  guide  the  design  of  generic  systems  to  optimize  human- 
machine  system  performance. 
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Systemes  adaptatifs  intelligents 

Revue  de  la  documentation  relative  a  /’orientation  de  la  conception 
de  I’automatisation  et  des  interfaces  adaptatives  intelligentes 


Resume 


II  est  possible  d’ameliorer  considerablement  les  performances  des  ensembles  homme- 
machine  en  ayant  recours  a  des  technologies  qui  peuvent  adapter  intelligemment 
l’interface  operateur-machine  (IOM)  et/ou  l’automatisation  des  taches  et  le  soutien 
accorde  a  l’operateur  conformement  au  contexte  exteme  (c.-a-d.  le  contexte  de  la 
tache)  et  au  contexte  interne  (c.-a-d.  l’etat  de  l’operateur).  Toutefois,  l’absence  de 
lignes  directrices  etablies  en  matiere  de  conception  constitue  un  lourd  obstacle  a  la 
conception  efficiente  de  systemes  adaptatifs  intelligents  (SAI).  Un  examen  approfondi 
de  la  documentation  a  ete  effectue  afin  d’examiner  les  demarches  actuelles  en 
conception  des  SAI  et  un  cadre  de  travail  unifie  a  ete  elabore  afin  de  decrire  des 
perspectives  conceptuelles  en  faisant  appel  a  une  terminologie  uniforme  et  non 
ambigue.  Par  ailleurs,  en  combinant  des  methodes  de  conception  des  domaines  des 
interactions  homme-ordinateur  (IHO)  et  des  facteurs  humains  (FH),  nous  avons 
elabore  des  cadres  conceptuels  et  de  design  afin  d’elaborer  des  lignes  d’orientation 
afin  d’aider  a  la  conception  et  a  la  mise  en  oeuvre  de  SAI.  Un  certain  nombre  de 
criteres  de  selection  des  methodes  analytiques  et  conceptuelles  appropriees  ont  aussi 
ete  developpes.  Les  cadres  recommandes  ne  guideront  pas  seulement  la  conception 
des  SAI  dans  les  domaines  militaires,  ils  aideront  aussi  dans  le  domaine  des  systemes 
civils  afin  d’optimiser  les  performances  des  systemes  homme-machine. 
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Executive  Summary 


Human-machine  system  performance  can  be  improved  by  utilising  technologies  that 
intelligently  adapt  the  operator  machine  interface  (OMI)  or  by  task  automation  and  support 
provided  to  the  operator  with  external  context  (i.e.,  task  environment)  and  internal  context 
(i.e.,  operator  state).  It  is  challenging  to  design  effective  Intelligent  Adaptive  Systems  (IASs) 
due  to  a  lack  of  established  design  guidelines.  Intelligent  Adaptive  Systems  assist  operators 
with  mental  and  physical  activities  such  as  decision  support  systems,  automation,  expert 
systems,  and  information  fusion.  IAS  provides  varying  degrees  of  autonomy;  they  can  be  used 
as  a  tools,  aids,  associates,  or  autonomous  agents.  IAS  act  as  assistants,  associates  or  coaches, 
providing  different  levels  of  sophistication.  In  addition,  a  lack  of  integration  between  the 
Human  Computer  Interaction  (HCI)  and  Human  Factors  (HF)  communities  has  allowed 
terminology  to  become  ambiguous  and  misleading  when  applied  globally.  There  is  a  pressing 
need  to  develop  a  framework  to  describe  these  approaches  using  consistent  and  unambiguous 
terminology.  There  is  a  paucity  of  established  design  guidelines  for  the  development  of 
advanced  operator  machine  interfaces  to  support  operators  in  dynamic,  error-critical,  and 
information-rich  domains. 

The  goal  of  this  literature  search  was  to  provide  support  to  establish  design  guidelines  for 
IASs  through  the  review  of  frameworks,  analysis  tools  and  processes  for  IASs,  and  through 
the  provision  of  general  design  recommendations  and  guidelines  for  the  development  of  IASs, 
with  particular  emphasis  on  the  OMI.  This  work  was  completed  under  contract  W771 1- 
067983,  to  DRDC  Toronto. 

Relevant  literature  was  collected  from  scientific,  defence,  government,  and  internet-based 
sources  pertaining  to  IASs.  All  articles  were  classified  in  terms  of  Level  of  Experimentation, 
Peer  Review,  Domain  Relevance  and  Literature  Review  Area.  The  literature  was  collated  and 
reduced  according  to  selection  criteria.  Each  article  was  evaluated  according  to:  the  degree  of 
peer  review  (e.g.,  technical  report,  conference  proceedings,  journal  article),  degree  of 
experimentation  involved  (e.g.,  conceptual  study  involving  no  experimentation,  laboratory- 
based  experimentation,  field-based  studies),  and  proximity  to  military  domains.  Each  article 
was  earmarked  for  inclusion  into  or  exclusion  from  the  final  review. 

After  reviewing  the  approaches  concerned  with  the  design  of  an  intelligent  adaptive  system,  a 
generic  conceptual  architecture  was  developed.  This  architecture  has  not  been  validated  as  of 
yet  and  has  the  following  four  components,  which  are  common  to  all  developed  and 
developing  IASs: 

•  Situation  Assessment  and  Support  System.  Involves  functionality  relating  to  real-time 
mission  analysis,  automation,  and  decision  support.  Monitors  and  tracks  the  current 
mission  state  and  aircraft/vehicle/system  status  (e.g.,  heading,  altitude,  threats  etc.) 
using  extensive  a  priori  task,  goal,  tactical,  operational,  and  situational  knowledge. 
Overall,  this  provides  information  about  the  state  of  the  aircraft/vehicle/system  within 
the  context  of  a  specific  mission,  and  uses  a  knowledge-based  system  to  provide 
assistance  (e.g.,  automate  tasks)  and  support  to  the  operator. 

•  Operator  State  Assessment.  Consists  of  functionality  relating  to  real-time  analysis  of 
the  psychological,  physiological  and/or  behavioural  state  of  the  operator.  Primary 


5 


functions  may  include  continuous  monitoring  of  workload,  inferences  about  current 
attention  focus,  ongoing  cognition  (e.g.,  visual  and  verbal  processing  load),  and 
intentions  using  extensive  a  priori  operator  knowledge  (e.g.,  models  of  human 
cognition,  control  abilities,  and  communication).  The  system  is  also  able  to  monitor 
the  operator  for  dangerously  high  and  low  levels  of  arousal.  Overall,  this  provides 
information  about  the  objective  and  subjective  state  of  the  operator  within  the  context 
of  a  specific  mission.  This  information  is  used  to  optimize  operator  performance  and 
safety,  and  provides  a  basis  for  the  implementation  of  pilot  assistance  and  support. 

•  Adaptation  Engine.  Uses  the  higher-order  outputs  from  Operator  State  Assessment 
and  Situation  Assessment  systems,  as  well  as  other  relevant  aircraft/vehicle/system 
data  sources,  to  increase  the  quality  of  the  match  between  aircraft/vehicle/system 
state,  operator  state,  and  the  tactical  assessments  provided  by  the  Situation 
Assessment  system.  These  integrative  functions  (operator  and  environmental  state) 
require  that  the  system  be  able  to  influence  the  prioritization  of  tasks  (i.e.,  intelligent 
adaptive  automation)  and/or  determine  the  means  by  which  information  is  presented 
to  the  operator  (i.e.,  intelligent  adaptive  interface). 

•  Operator  Machine  Interface.  The  means  by  which  the  operator  interacts  with  the 
aircraft/vehicle/system  in  order  to  fulfil  mission  tasks  and  goals.  This  is  also  the 
means  by  which,  the  operator  interacts  with  an  intelligent  adaptive  system  (e.g.,  a 
tasking  interface  manager).  The  design  of  the  OMI,  is  defined  by  existing  HF  and 
HCI  best-practice  and  standards. 

All  four  components  operate  within  the  context  of  a  closed-loop  system:  a  feedback  loop  re¬ 
samples  operator  state  and  situation  assessment  following  the  adaptation  of  the  OMI  or 
automation.  The  goal  is  to  adjust  the  level  of  adaptation  so  that  optimal  operator  states  (e.g., 
performance,  workload  etc)  are  attained  and  maintained. 

The  literature  research  achieved  the  following  goals: 

•  Identified  the  advantages,  disadvantages  and  applicability  of  development 
frameworks,  analysis  methodologies,  design  approaches,  and  operator-state 
monitoring  approaches; 

•  Make  some  progress  in  unifying  independent  HF  and  HCI  approaches  to  the 
development  of  IASs  by  providing  a  generic  framework  that  maps  to  both  approaches 
by  focusing  on  system  functionality  and  capability;  and, 

•  Integrate  design  methodologies  from  both  HCI  and  HF  fields  and  develop  guidance 
for  developers  to  assist  in  the  design,  development  and  implementation  of  IASs,  with 
emphasis  on  development  of  the  OMI.  A  number  of  criteria  for  the  selection  of 
appropriate  analytical  and  design  approaches  were  also  recommended. 

The  proposed  frameworks  will  not  only  provide  guidance  for  designing  IASs  in  military 
domains,  but  will  also  guide  other  civilian  systems  to  optimize  human-machine  system 
performance. 
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Systemes  adaptatifs  intelligents 

Revue  de  la  documentation  relative  a  /’orientation  de  la  conception 
de  I’automatisation  et  des  interfaces  adaptatives  intelligentes 


SOMMAIRE 

II  est  possible  d’ameliorer  considerablement  les  performances  des  ensembles  homme- 
machine  en  ayant  recours  a  des  technologies  qui  peuvent  adapter  intelligemment 
l’interface  operateur-machine  (IOM)  et/ou  l’automatisation  des  taches  et  le  soutien 
accorde  a  l’operateur  conformement  au  contexte  exteme  (c.-a-d.  le  contexte  de  la 
tache)  et  au  contexte  interne  (c.-a-d.  l’etat  de  l’operateur).  Toutefois,  l’absence  de 
lignes  directrices  etablies  en  matiere  de  conception  constitue  un  lourd  defi  a  la 
conception  efficiente  de  systemes  adaptatifs  intelligents  (SAI).  Ces  systemes  aident 
les  operateurs  pour  une  multitude  d’activites  mentales  et  physiques  (p.  ex.  les 
systemes  d’aide  a  la  decision,  l’automatisation,  les  systemes  experts  et  la  fusion 
d’information),  avec  divers  degres  d’autonomie  (p.  ex.,  outils,  aides,  associes,  agents 
autonomes)  et  de  perfectionnement  (p.  ex.,  assistant,  associe  ou  conseiller).  En  outre, 
l’integration  insuffisante  des  collectivites  s’interessant  aux  interactions  homme- 
machine  (IHO)  et  aux  facteurs  humains  (HF)  a  favorise  le  developpement  de 
terminologies  de  plus  en  plus  ambigiies,  ou  meme  trompeuses,  lorsqu’elles  sont 
appliquees  de  maniere  generate.  II  devient  done  pressant  de  developper  un  cadre  de 
travail  unifie  afin  de  decrire  des  perspectives  conceptuelles  en  faisant  appel  a  une 
terminologie  uniforme  et  non  ambigue.  Par  ailleurs,  il  n’y  a  pas  de  lignes  directrices 
etablies  en  matiere  de  conception  pour  le  developpement  d’ interfaces  operateur- 
machine  avancees  afin  d’aider  les  operateurs  dans  les  domaines  dynamiques,  propices 
a  l’occurrence  d’erreurs  critiques  et  riches  en  information. 

L’objectif  de  cette  revue  de  la  documentation  consistait  a  apporter  un  soutien  a 
l’etablissement  de  lignes  directrices  pour  la  conception  de  systemes  adaptatifs 
intelligents  en  etudiant  les  cadres  de  travail,  les  outils  et  processus  d’ analyse  servant 
les  SAI,  et  en  elaborant  des  recommandations  et  lignes  directrices  generates  pour  la 
conception  et  le  developpement  de  SAI,  avec  une  insistance  particuliere  sur  les  IOM. 
Ce  travail  a  ete  effectue  dans  le  cadre  d’un  contrat  conclu  avec  RDDC  Toronto. 

La  documentation  pertinente  a  ete  recueillie  a  partir  de  sources  scientifiques,  de  la 
defense,  du  gouvemement  et  d’lntemet  portant  sur  les  SAI.  Tous  les  articles  ont  ete 
classes  en  fonction  du  niveau  d’ experimentation,  de  Texamen  par  les  pairs,  de  la 
pertinence  au  domaine  et  du  domaine  d’examen  de  la  documentation.  La 
documentation  a  ete  colligee  et  reduite  en  fonction  de  criteres  de  selection  appropries. 
Chaque  article  a  ete  evalue  en  fonction  du  degre  d’examen  par  les  pairs  (p.  ex., 
rapport  technique,  actes  de  conference,  articles  de  joumaux),  du  degre 
d’ experimentation  realisee  (p.  ex.,  etude  conceptuelle  n’impliquant  aucune 
experimentation,  experimentation  en  laboratoire,  etudes  sur  le  terrain)  et  proximite  des 
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domaines  militaires.  Chaque  article  a  aussi  ete  classe  en  consequence  en  vue  d’etre 
inclus  dans  le  processus  d’examen  final,  ou  exclus  de  ce  processus. 

Apres  avoir  examiner  les  demarches  visant  le  developpement  d’un  systeme  adaptatif 
intelligent,  nous  avons  elabore  une  architecture  conceptuelle  generique.  Cette  demiere 
se  compose  de  quatre  modules  communs  a  tous  les  SAI  developpes  et  en  voie  de 
developpement : 

•  Systeme  d’ evaluation  et  de  soutien  de  la  situation.  Cet  element  comporte  des 
fonctionnalites  liees  a  l’analyse  de  mission,  a  l’automatisation  et  a  la  prise  de 
decision  en  temps  reel.  Ce  systeme  effectue  la  surveillance  et  le  suivi  de  l’etat 
d’une  mission  en  cours  et  l’etat  des  aeronefs,  vehicules  ou  systemes  (p.  ex., 
cap,  altitude,  menaces,  etc.)  en  faisant  appel  a  des  connaissances  prealables 
etendues  des  taches  et  objectifs,  ainsi  que  des  aspects  tactiques,  operationnels 
et  situationnels.  Dans  l’ensemble,  tout  cela  foumit  de  l’information  au  sujet  de 
l’etat  objectif  des  aeronefs,  vehicules  ou  systemes  dans  le  contexte  d’une 
mission  specifique  et  il  fait  appel  a  un  systeme  a  base  de  connaissances  pour 
preter  assistance  (p.  ex.,  en  automatisant  des  taches)  et  preter  appui  a 
l’operateur. 

•  Evaluation  de  l  ’etat  de  l  'operateur.  Cet  element  comporte  des  fonctionnalites 
bees  a  l’analyse  en  temps  reel  de  l’etat  psychologique,  physiologique  et/ou 
comportemental  de  1’ operateur.  Les  principales  fonctions  peuvent  comprendre 
la  surveillance  en  continu  de  la  charge  de  travail,  des  inferences  au  sujet  de  la 
concentration,  des  connaissances  perceptuelles  (p.  ex.,  charge  de  traitement 
visuelle  et  verbale)  et  des  intentions  en  faisant  appel  a  des  connaissances 
prealables  de  l’operateur  (p.  ex.,  modeles  cognitifs  humains,  habilites  de 
commande  et  communication).  Ce  systeme  est  aussi  en  mesure  de  surveiller 
l’operateur  pour  detecter  les  niveaux  dangereusement  eleves  ou  bas  d’eveil. 
Dans  l’ensemble,  cela  foumit  de  l’information  au  sujet  de  l’etat  subjectif  et 
objectif  de  l’operateur  dans  le  contexte  d’une  mission  specifique.  Cette 
information  est  utilisee  afin  d’optimiser  le  rendement  et  la  securite  de 
l’operateur,  et  cela  foumit  une  base  pour  la  mise  en  oeuvre  de  systemes  d’aide 
et  de  soutien  des  pilotes. 

•  Moteur  adaptatif.  II  fait  appel  aux  extrants  de  niveau  eleve  du  module 
devaluation  d’etat  de  l’operateur  et  devaluation  et  de  soutien  de  la  situation, 
ainsi  que  d’autres  sources  de  donnees  pertinentes  des 

aeronefs/vehicules/systemes  afin  de  maximiser  la  correspondance  entre  l’etat 
des  aeronefs/vehicules/systemes,  l’etat  de  l’operateur  et  les  evaluations 
tactiques  foumies  par  le  systeme  devaluation  de  la  situation.  Ces  fonctions 
integratives  exigent  que  le  systeme  soit  capable  d’influer  sur  la  priorisation  et 
l’affectation  des  taches  (c.-a-d.  l’automatisation  adaptative  intelligente)  et/ou 
de  determiner  les  moyens  de  presentation  de  l’information  a  l’operateur  (ce  qui 
correspond  a  l’interface  adaptative  intelligente). 
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•  Interface  operateur-machine.  C’est  le  dispositif  qui  permet  a  l’operateur 
d’interagir  avec  les  aeronefs/vehicules/systemes  afm  de  satisfaire  les  taches  et 
objectifs  d’une  mission.  II  s’agit  aussi  d’un  moyen  qui  permet,  le  cas  echeant, 
a  l’operateur  d’interagir  avec  le  systeme  adaptatif  intelligent  (p.  ex.,  un 
gestionnaire  d’interface  d’attribution  de  tache).  La  conception  de  1’IOM,  ainsi 
que  son  automatisation,  sont  definies  par  les  pratiques  exemplaires  et  normes 
reconnues  dans  le  domaine  des  FH  et  des  IHO. 

Les  quatre  modules  fonctionnent  dans  une  structure  a  boucle  fermee  :  une  boucle  de 
reaction  echantillonne  l’etat  de  l’operateur  et  V evaluation  de  la  situation  apres 
l’adaptation  de  1’IOM  et/ou  F automatisation.  Ce  processus  a  pour  objectif  d’ajuster  le 
niveau  d’ adaptation  afm  que  des  niveaux  optimaux  puissent  etre  atteints  et  maintenus 
pour  l’operateur  (p.  ex.,  sur  le  plan  des  performances,  de  la  charge  de  travail,  et  ainsi 
de  suite). 

L’examen  de  la  documentation  a  permis  d’atteindre  les  resultats  suivants  : 

•  Relever  les  avantages  et  desavantages  et  etablir  1’ applicability  des  cadres  de 
developpement,  des  methodes  d’ analyse  et  de  conception,  ainsi  que  les  methodes  de 
surveillance  de  l’etat; 

•  Progresser  dans  l’unification  des  approches  jusqu’a  present  independant  en  matiere  de 
facteurs  humains  et  d’lHO  pour  le  developpement  de  SAI  en  fournissant  une 
architecture  conceptuelle  generique  qui  correspondrait  aux  deux  approches  en  se 
concentrant  sur  les  fonctionnalites  et  capacites; 

•  Integrer  des  methodes  de  conception  des  domaines  des  IHO  et  des  facteurs  humains  et 
elaborer  des  lignes  d’orientation  a  l’intention  des  developpeurs  afin  d’aider  a  la 
conception,  au  developpement  et  a  la  mise  en  oeuvre  de  SAI,  en  mettant  un  accent 
particulier  sur  le  developpement  des  IOM.  Un  certain  nombre  de  criteres  de  selection 
des  methodes  analytiques  et  conceptuelles  appropriees  ont  aussi  ete  recommandes. 

Les  cadres  recommandes  ne  guideront  pas  seulement  la  conception  des  SAI  dans  les  domaines 
militaires,  ils  aideront  aussi  dans  le  domaine  des  systemes  civils  afin  d’optimiser  les 
performances  des  systemes  homme-machine. 
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1  Introduction 


The  Department  of  National  Defence,  Defence  Research  and  Development  Canada  (DRDC), 
Toronto,  Ontario,  has  started  a  research  project  to  examine  the  application  of  theoretical 
approaches  to  the  design  of  intelligent  and/or  adaptive  interfaces  and  to  develop  design 
recommendations  that  maximize  overall  human-machine  system  performance.  Currently, 
there  are  few  established  design  guidelines  for  advanced  operator  interfaces  that  provide 
assistance  to  decision  makers  who  need  to  manage  the  vast  amounts  of  data  and  information 
in  a  complex  and  networked  environment.  DRDC  Toronto  has  completed  a  project  focussing 
on  the  design  and  development  of  intelligent  adaptive  interfaces  (IAI)  in  the  context  of 
controlling  multiple  Uninhabited  Aerial  Vehicles  (UAVs).  The  aim  of  this  UAV  IAI  project 
was  to  develop  operator  interface  design  guidelines  to  support  reduced  manning  and  enhanced 
performance  in  complex  military  systems.  Within  this  project,  theoretical  frameworks  and 
design  concepts  for  designing  an  agent-based  system  were  developed.  Under  this  theoretical 
guidance,  hierarchical  goal  analysis  and  performance  modelling  were  conducted  to  compare 
overall  system  performance  while  controlling  multiple  UAVs  with  and  without  the  aid  of 
automation  agents.  Preliminary  design  guidelines  were  identified. 

The  goal  of  this  literature  review  is  to  provide  support  to  establish  design  guidelines  for 
Intelligent  Adaptive  Systems  (IASs)  through  the  review  of  frameworks,  analysis  tools  and 
processes  for  IASs,  and  through  the  provision  of  general  design  recommendations  and 
guidelines  for  the  development  of  IASs,  with  particular  emphasis  on  the  Operator  Machine 
Interface  (OMI).  This  work  was  completed  under  contract  W771 1-067983,  to  DRDC  Toronto. 

1.1  Intelligent  Adaptive  Systems:  Automation  and  Interface 

Modern  technology  is  very  complex;  this  allows  vast  amounts  of  data  to  be  available  to 
operators  from  many  sources.  Considerable  computerized  assistance  is  needed  for  operators  to 
be  able  to  integrate  and  act  upon  the  data.  However,  perceiving  and  interpreting  all  of  the 
relevant  information  and  choosing  an  appropriate  response  within  the  temporal  constraints  of 
the  situation  would  challenge  any  intelligent  agent,  human  or  machine  (Banbury,  Bonner, 
Dickson,  Howells  and  Taylor,  1999). 

Traditionally,  there  have  been  two  main  thrusts  of  research  and  development  undertaken  to 
address  problems  associated  with  operators  working  under  conditions  of  excessive  workload 
levels  (e.g.,  sub-optimal  task  performance,  error,  loss  of  Situation  Awareness  [SA],  etc.).  The 
first  approach  originated  from  the  Human  Factors  (HF)  community,  in  which  research  was 
conducted  to  study  the  effects  of  adaptable  automation1  on  operator  performance  and 
workload  within  error-critical  domains,  such  as  aviation  and  industrial  process  control.  The 
second  approach  originated  from  the  Human  Computer  Interaction  (HCI)  community,  in 


1  Automation:  The  mechanisation  and  integration  of  the  sensing  of  environmental  variables;  machine 
data  processing  and  decision  making;  machine  mechanical  action;  and,  machine  information 
action/communication  (Sheridan  and  Parasuraman,  2006). 
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which  research  was  conducted  to  study  the  effects  of  adaptable  operator  machine  interfaces2 
on  operator  performance  within  relatively  more  benign  domains,  such  as  word  processing  and 
web  browsing. 

Adaptable  automation  and  OMIs  address  very  similar  issues;  both  adaptable  automation  and 
OMIs  seek  to  reduce  operator  workload,  and  in  doing  so,  facilitate  more  efficient  task 
performance  by  replacing  or  augmenting  the  human  operator.  In  addition,  both  rely  on  a  user 
model  to  adapt  the  system  to  pre-deflned  operator  characteristics  (e.g.,  workload,  task 
performance  and  progress,  and  so  on)  according  to  the  status  of  a  task  model  (e.g.,  required 
tasks/goals).  Intelligent  Adaptive  Systems,  therefore,  seek  to  enhance  human-machine  system 
performance  by  utilising  technologies  that  can  intelligently  adapt  the  OMI  and/or  task 
automation  and  support  provided  to  the  operator  in  accordance  with  the  both  the  external 
context  (i.e.,  task  environment)  and  internal  context  (i.e.,  operator  state).  The  human  operator 
is  an  intrinsic  part  of  the  IAS,  given  the  closed-loop  nature  of  IASs  (e.g.,  monitoring  and 
adapting  to  operator  state  changes  the  nature  of  how  tasks  are  performed,  which  in  turn 
requires  re-sampling  of  operator  state),.  The  terminology  used  in  this  report  should  therefore 
be  defined  at  the  outset:  human  refers  to  the  operator,  machine  refers  to  the  device  used  to 
perform  a  task  or  assist  the  operator  perform  the  task,  and  system  refers  to  the  synergy  of  the 
two. 

Despite  the  obvious  similarity  between  the  HF  and  HCI  research  in  intelligent  adaptive 
systems,  there  is  little  research  that  integrates  these  two  research  streams.  This  is  an 
unfortunate  oversight  by  the  HF  and  HCI  communities  as  insufficient  integration  between  HF 
and  HCI  could  increase  the  potential  for  confusion  in  terminology.  For  example,  many  of  the 
intelligent  adaptive  automation  systems  in  development  involve  some  degree  of  OMI 
adaptation  (e.g.,  the  Tasking  Interface  Manager  of  the  Cognitive  Cockpit;  see  Section  10.2.3). 
However,  this  research  has  not  drawn  upon  the  findings  from  the  HCI  research  studies  on 
adaptable  OMIs,  primarily  since  the  HCI  field  has  concentrated,  almost  exclusively,  on 
computing  applications,  such  as  word-processing  and  web-browsing.  Similarly,  key  studies 
from  the  HF  community  are  rarely  cited  by  the  HCI  community  (see  Section  6.4.3).  Thus,  the 
terminology  used  by  both  fields  to  describe  identical  systems  is  different.  The  remainder  of 
this  section  focuses  on  briefly  identifying  similarities  between  the  HF  and  HCI  approaches, 
and  developing  consistent  terminology  that  describes  the  functions  and  capabilities  of  IASs 
that  map  to  both  HF  and  HCI  approaches. 

Figure  1  draws  parallels  between  the  historical  origins  of  the  development  of  Intelligent 
Adaptive  Interfaces  within  the  HCI  community  and  Intelligent  Adaptive  Automation  (IAA) 
within  the  HF  community,  resulting  in  Intelligent  Adaptive  Hybrid  (I AH)  systems.  This 
reflects  the  use  of  both  adaptable  automation  and  an  adaptable  interface  within  the  same 
system.  Each  system  will  be  considered  as  IASs  (depicted  by  the  blue  shaded  area)  and  will 
be  reviewed  within  this  report.  The  technologies  described  in  Figure  1  are  not  intended  to 
represent  discrete  stages  in  development,  but  instead,  represent  steps  upon  a  continuum  of 
intelligent  adaptive  system  development. 


2  Operator  Machine  Interface:  The  aggregate  of  means  by  which  the  human  operator  interacts  with  the 
machine.  The  OMI  provides  the  means  of  input  (i.e.,  allowing  the  operator  to  manipulate  the  machine) 
and  output  (i.e.,  allowing  the  machine  to  produce  the  effects  of  the  operator’s  manipulation). 
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Figure  1:  Illustration  of  the  parallel  development  of  Intelligent  Adaptive  Systems  from 

the  HCI  and  HF  communities. 


Conventional  Automation  (i.e.,  the  replacement  or  augmentation  of  human  work  with 
mechanical  or  electronic  machines,  such  as  an  autopilot)  and  the  conventional  operator- 
machine  Interface  (i.e.,  a  common  boundary  shared  by  a  human  operator  and  a  machine, 
across  which  data  or  information  flows,  such  as  a  Windows-based  display)  are  both  designed 
using  a  combination  of  user  requirements  (i.e.,  functionality  and  capability  required  to 
perform  a  given  task),  and  user  preferences  (i.e.,  functionality  and  capability  that  is  not 
necessarily  required  to  perform  a  give  task,  but  nonetheless  improves  the  perceived  quality  of 
the  operator’s  interaction  with  the  machine).  In  addition,  automation  capability  is  also  based 
on  the  ‘left-over’  principle:  operators  are  left  with  functions  that  have  not  been  automated  or 
could  not  be  automated,  and  the  ‘compensatory’  principle:  functions  are  allocated  according 
to  the  strengths  and  weaknesses  of  the  human  and  the  machine  (e.g.,  Fitts,  1951). 

The  progression  from  conventional  automation  and  interfaces  to  Adaptive  Automation  and 
Adaptive  Interfaces  is  possible  through  the  use  of  a  Task  Model  (i.e.,  the  system’s  information 
of  the  task  activities  that  are  likely  to  be  conducted  by  the  operator).  Similarly,  the 
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progression  from  these  systems  to  Intelligent  Adaptive  Automation  and  Intelligent  Adaptive 
Interfaces  is  possible  through  the  use  of  a  User  Model  (i.e.,  the  system’s  knowledge  of  the 
capabilities,  limitations  and  knowledge  of  the  human  operator),  in  addition  to  the  task  model. 
Finally,  the  combination  of  automation  and  interface  adaptation,  Intelligent  Adaptive  Hybrid 
systems,  is  depicted  as  the  confluence  of  the  two  research  streams.  Sections  6.2,  6.3  and  6.4 
describe  each  of  these  seven  technologies  in  more  detail. 


1 .2  Origins  of  Intelligent  Adaptive  Interfaces 

Operator  machine  interface  technologies  exist  in  a  number  of  guises,  from  conventional 
interfaces,  to  adaptive  interfaces,  and  finally  to  intelligent  adaptive  interfaces.  Sections  6.2.1 
through  6.2.3  describe  the  evolution  of  intelligent  adaptive  interfaces. 


1.2.1  Conventional  (Operator-Machine)  Interfaces 

In  their  most  general  form,  conventional  operator-machine  interfaces  are 
the  medium  that  supports  operator  interaction  with  a  particular  machine, 
device,  computer  program  or  other  complex  tool.  The  OMI  facilitates 
input,  allowing  the  operator  to  manipulate  a  system,  and  output,  allowing 
the  machine  to  produce  the  effects  of  the  operator’s  manipulation. 


1.2.2  Adaptive  Interfaces 

Adaptive  interfaces  enhance  operator  interaction  with  a  system  by  making 
the  system  more  efficient,  effective  and  easy  to  use.  The  interface  is  adapted 
(e.g.,  menu  content)  with  the  aim  of  matching  its  content  to  changing  task- 
related  circumstances  (e.g.,  according  to  the  mode  selected,  application 
used,  etc.).  The  system  controls  the  adaptation  of  the  interface,  and  how  it 
occurs,  along  with  the  amount  of  adaptation  that  occurs;  although,  the 
operator  has  some  control  over  how  the  system  adaptation  is  configured 
initially. 


5|yin(  «iit  LVArrirw...  Pi 

a 

giKAwh.. .  fifr  ■  flrk 

Lhwp  ► 

WDfdUKmt... 

sp«th 

a-u-mJ  WbrtetMin. .. 

1  ml  Hflinfp  ■ 

wt™  ■ 

jjj 

1 .2.3  Intelligent  Adaptive  Interfaces 

Intelligent  Adaptive  Interfaces  are  OMIs  that  change  their  control  and/or 
display  characteristics  to  react  to  task  and/or  operator  states  in  real-time 
(Hou,  2007).  Intelligent  Adaptive  Interfaces  are  epitomised  by  Microsoft’s 
Office  Assistant.  This  feature  was  included  in  Microsoft  Office  97  and 
subsequent  versions  until  Office  2007.  The  most  ‘popular’  Office  Assistant 
was  named  “Clippy”  after  its  default  animated  paperclip  representation. 

This  feature  was  an  entry  point  to  the  application’s  help  system,  presenting  various  help 
search  functions  and  offering  advice  based  on  Bayesian  algorithms.  Clippy  would  open  when 
the  program  thought  the  operator  required  assistance,  and  would  modify  the  formatting  of  the 
document  and  content  of  the  menus  accordingly.  For  example,  typing  an  address  followed  by 
“Dear”  would  prompt  Clippy  to  open  and  state  “It  looks  like  you’re  writing  a  letter.  Would 
you  like  help?”.  The  algorithms  would  use  a  combination  of  task-based  (e.g.,  how  a  letter  is 
usually  formatted)  and  user-based  (e.g.,  how  many  mistakes  the  operator  has  made  trying  to 
write  a  letter)  models  to  modify  the  interface  to  match  the  operator’s  needs  and  requirements. 
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Similar  to  adaptive  interfaces,  the  amount  of  adaptation  that  occurs  is  controlled  entirely  by 
the  system,  although  the  operator  has  some  control  over  how  the  system  adaptation  is 
configured  initially  (e.g.,  the  user  is  capable  of  turning  off  Clippy).  Intelligent  Adaptive 
Interface  systems  are  reviewed  in  Section  8.2  of  this  report. 


1 .3  Origins  of  Intelligent  Adaptive  Automation 

The  advent  of  automation  technology  has  created  the  opportunity  to  decrease  the  tasks 
operators  perform  and  assist  them  in  making  tactical  and  strategic  decisions.  While 
conventional  automation  has  often  been  designed  to  replace  human  control  and  decision 
making,  intelligent  aiding  seeks  to  augment  and  enhance  human  judgment  and  responsibility. 
Sections  6.3.1,  6.3.2,  and  6.3.3  describe  the  evolution  of  intelligent  adaptive  automation. 


1.3.1  Conventional  (Static)  Automation 

The  general  intention  of  conventional,  or  static,  automation  is  to  increase 
safety  and  efficiency  by  moderating  operator  workload.  Advances  in 
automation  technology  have  facilitated  a  wide  variety  of  tasks  to  be 
completed  under  automatic  control,  such  as  flight  controls,  navigation 
systems,  and  system  health  checks.  The  conventional  approach  to  automation 
sought  to  replace  operator  involvement  in  certain  tasks  with  automated 
systems.  As  technology  advanced,  more  functions  were  considered  for 
automation;  the  general  principle  was  that  if  an  automated  system  could  surpass  human 
performance,  the  function  was  allocated  to  the  system.  This  approach  is  also  referred  to  as 
static  automation.  The  system  designer  initiates  an  allocation  of  function  between  the  operator 
and  the  automated  system  (i.e.,  the  agent  in  control  is  fixed  for  the  duration  of  the  task).  The 
level  of  automation  is  not  context  dependent  and  therefore  is  not  sensitive  to  the  external 
situation. 


The  allocation  of  tasks  between  operator  and  machine  can  range  from  full  automation  to  full 
operator  control.  Task  allocation  depends  on  system  measures  and  task  efficiency,  rather  than 
on  satisfying  the  needs  of  the  human  operator.  For  example,  Fitts  (1951)  created  a  set  of 
criteria  where  the  human  can  exceed  machine  capability  (e.g.,  the  ability  to  improvise  and  use 
flexible  procedures,  and  the  ability  to  reason  inductively),  and  where  the  machine  can  surpass 
human  capability  (e.g.,  the  ability  to  perform  repetitive  and  routine  tasks) 

However,  such  fixed  allocation  of  function  has  a  tendency  to  ignore  the  fundamental  question: 
how  do  changes  to  the  allocation  of  tasks  affect  operator  performance?  Effects  on  the 
cognitive  abilities  and  deficiencies  of  the  operator  (e.g.,  motivation,  tension,  boredom  and 
fatigue)  are  of  paramount  importance  to  the  safe  operation  of  the  overall  human-in-the-loop 
system.  For  example,  whilst  the  machine  can  outperform  an  operator  on  most  monitoring 
tasks,  the  human  excels  in  the  ability  to  adapt  when  situations  change. 

There  are  concerns  that  this  traditional  approach  has  not  been  entirely  successful  and  that 
increased  automation  may  make  an  operator’s  task  more  difficult.  Much  research  (see  section 
10.1.1)  has  demonstrated  problems  associated  with  conventional  automation  in  aviation, 
including:  increased  monitoring  load;  out-of-the-loop  performance  problems;  loss  of  skills; 
complacency;  lack  of  trust  (i.e.,  scepticism);  and  increased  system  complexity.  Challenges  for 
operators  using  conventional  automation  can  be,  maintaining  direct  involvement  in  the  task 
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and  interpreting  large  amounts  of  information.  This  can  result  in  a  shift  from  an  operator 
being  a  hands-on  controller,  towards  the  role  of  a  being  system  manager  and  system  monitor. 
This  is  an  unfortunate  development  considering  that  humans  are  not  reliable  monitors. 
Humans  may  experience  difficulty  maintaining  alertness  and  vigilance  over  time  without 
active  involvement  in  the  system’s  operation. 


1 .3.2  Adaptive  Automation 

The  desire  to  improve  the  relation,  and  therefore  performance, 
between  human  and  machine  has  prompted  the  development 
of  an  alternative  approach  to  automation.  Adaptive  automation 
is  described  as  ‘adaptive’  because  the  control  of  the 
commencement  and  the  ending  of  specific  tasks  are  shared 
between  the  operator  and  the  machine.  Adaptive  automation 
is  required  to  minimise  the  negative  effects  of  static  allocation 
(e.g.,  skill  degradation  and  reduction  of  situation  awareness), 
while  simultaneously  achieving  optimal  levels  of  system  performance.  An  example  of  a 
simplistic  adaptive  automation  system  is  a  modem  flight  management  system  that  automates 
the  presentation  and  partial  completion  of  operating  checklists  according  to  the  phase  of 
flight,  or  the  detection  of  sub-system  failures  (i.e.,  using  a  task  model).  More  complex  forms 
of  adaptive  automation  utilise  simplistic  user  models  (e.g.,  simple  behavioural  indices  of  high 
workload)  as  a  mechanism  to  control  the  onset  and  offset  of  automation. 


Adaptive  automation  represents  an  alternative  design  approach  to  the  implementation  of 
automation  in  that  the  relationship  between  machine  and  operator  is  flexible  and  context 
dependent.  The  provision  of  adaptive  machine  aiding  is  not  pre-determined  at  the  design 
stage.  The  task  allocation  to  the  human  or  system  is  not  fixed.  An  adaptive  automation  system 
is  dynamic  in  nature  in  that  the  loci  of  control,  the  control  function  within  a  system,  are 
constantly  changing. 


Adaptive  automation  seeks  to  take  advantage  of  the  differences  between  the  abilities  of 
humans  and  machines  through  a  strategy  that  allows  changes  in  task  allocation.  An  example 
of  a  strategy  for  adaptive  task  allocation  would  be  operator  workload  levels.  Emphasis  is 
given  to  the  prevention  of  task  over-load  (or  under-load)  imposed  on  the  operator,  at  defined 
point(s)  along  a  continuum. 


Adaptive  automation  occurs  when  the  control  decisions  concerning  the  onset,  offset,  and 
degree  of  automation  are  shared  between  the  operator  and  machine.  Within  such  a  system,  the 
human  operator  remains  ‘in-the-loop’  and  the  automation  intervenes  only  when  an  increase 
(or  decrease)  in  operator  workload  requires  system  support  to  meet  operational  requirements. 
In  providing  this  dynamic  or  adaptive  support,  the  perceived  loss  of  control  associated  with 
static  automation  can  be  reduced. 


Studies  have  shown  (Hilburn,  Molloy,  Wong  &  Parasuraman,  1993;  Parasuraman,  Moluloua, 
Molly  &  Hilburn,  1993;  Riley  &  Parasuraman,  1997)  that  the  detection  of  automation  failures 
is  substantially  degraded  in  systems  with  conventional  automation  in  which  the  allocation  of 
tasks  between  operator  and  system  remains  fixed  over  time  (see  section  10.1.1).  When  using 
an  adaptive  automation  system,  brief  periods  of  manual  task  allocation  increases  the  detection 
rate  of  automation  failures  (Hilburn  et  al.  1993,  Parasuraman  et  al.  1993).  Adaptive 
automation  improved  monitoring  performance  over  relatively  long  periods  of  time.  However, 
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adaptive  automation  introduces  complexity  during  task  allocation  that  can  result  in  new 
problems  of  awareness  of  system  functional  state  and  automation  failure  detection.  A  key 
design  issue  is  optimising  triggering  conditions  for  task  re-allocation  (e.g.,  by  monitoring 
operator  behaviour  or  situation/task  events;  see  Section  8. 5. 1.1). 

Several  studies  have  compared  adaptive  to  conventional  automation,  or  manual  control.  They 
have  shown  that  the  introduction  of  adaptive  automation  systems  has  met  with  some  success. 
These  results  can  be  summarised  as  follows: 

•  Reductions  in  response  time  to  flight  management  task  demands  of  up  to  40%  (Chu 
and  Rouse,  1979); 

•  Reductions  in  time  to  place  sensors  in  anti-submarine  warfare  of  15%  (Freedy,  Madni 
and  Samet,  1985); 

•  Increases  of  5-9%  in  tracking  performance  together  with  increases  of  up  to  25%  in 
identification  performance  in  an  aerial  reconnaissance  task  (Morris  and  Rouse,  1986); 
and, 

•  Increases  of  up  to  25%  in  tracking  performance  together  with  increases  of  up  to  42% 
in  target  identification  in  a  reconnaissance  task  (Forester,  1986). 

1 .3.3  Intelligent  Adaptive  Automation 

Traditionally,  automation  design  decisions  have  focused  on 
optimising  the  performance  of  the  technology  (i.e., 
technology-centred).  IAA  is  human-centred;  IAA  design  is 
based  on  a  consideration  of  human  limitations  and 
capabilities,  rather  than  of  system  and  mission  performance. 

IAA  systems  rely  heavily  on  detailed  and  comprehensive  user 
models  (e.g.,  models  of  human  cognition,  knowledge  of 
human  capabilities  and  limitations).  These  systems  seek  to 
restore  the  pilot  to  the  role  of  the  decision-maker,  while  at  the 
same  time  providing  safeguards  for  situations  in  which  time  limitations,  or  the  complexity  of 
the  problem,  restrict  operator  problem  solving  ability. 

Intelligent  Adaptive  Automation  seeks  to  augment  and  enhance  an  operator’s  judgement  and 
responsibility,  while  mitigating  the  operator’s  limitations.  These  systems  can  be  considered  to 
be  ‘intelligent’  insofar  as  they  exhibit  behaviours  that  are  consistent  with  intelligent  human¬ 
like  characteristics  (Taylor  and  Reising,  1998,  1999),  such  as: 

•  Active  collection  of  information; 

•  Goal  driven; 

•  Capable  of  reasoning  at  multiple  levels;  and, 

•  Capable  of  learning  from  experience. 

Functional  integration,  rather  than  function  allocation,  is  an  important  characteristic  of  IAA 
systems  (Geddes,  1997).  As  tasks  become  more  mental  than  physical,  the  validity  of  applying 
the  concept  of  functional  separation  of  tasks  is  debatable.  With  functional  integration, 
behaviours  required  by  the  domain  are  shared  across  the  functional  components,  including  the 
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operator.  The  same  behaviour  can  be  performed  by  several  functional  components,  rather  than 
just  one,  providing  more  robust  and  flexible  integrated  systems  than  systems  in  which 
functions  are  allocated  to  specific  system  components. 

Previous  attempts  and  ongoing  development  of  Intelligent  Adaptive  Automation  systems  can 
be  distinguished  in  terms  of  the  tasks  and  roles  that  they  perform  (Geddes  and  Shalin,  1997): 

•  Assistant.  Performs  specific  tasks  when  instructed  by  the  operator,  using  basic  task 
and  situation  knowledge.  For  example,  a  system  could  provide  a  pilot  with  an 
assessment  of  a  threatening  aircraft  when  asked. 

•  Associate.  Automatically  recognises  that  the  operator  requires  assistance  (using 
complex  task  and  situation  knowledge,  and  basic  user  knowledge),  and  provides  some 
level  of  support.  For  example,  a  system  could  recognise  a  threatening  situation  and 
automatically  provide  the  pilot  with  all  threat  information. 

•  Coach.  Using  complex  task,  situation  and  user  knowledge,  these  types  of  systems  are 
capable  of  recognising  the  need  for  automation  in  order  to  achieve  a  mission 
objective,  and  providing  instructions  to  the  operator  on  how  to  achieve  the  objective. 
For  example,  the  pilot  is  presented  with  the  most  threatening  aircraft  first,  in 
accordance  with  the  higher-level  goal  of  maximising  own-ship  survivability. 

Intelligent  Adaptive  Automation  systems  are  reviewed  in  Section  8.3. 

1.4  Intelligent  Adaptive  Hybrid  Systems 

Intelligent  Adaptive  Hybrid  systems  are  a  combination  of 
IAA  and  IAI  technologies  that  are  capable  of  context- 
sensitive  communication  with  the  operator.  IAH  systems 
technologies  currently  under  construction  operate  at  the 
level  of  Assistant  (e.g.,  Germany’s  Cockpit  Assistant 
System  [CASSY]/  Crew  Assistant  Military  Aircraft 
[CAMMA]  programmes,  France’s  Co-pilote  Electronique 
programme),  Associate  (e.g.,  USAF  Pilots’  Associate  (PA) 
programme  and  US  Army  Rotorcraft  Pilots’  Associate 
(RPA)  programme),  and  Coach  (e.g.,  the  United  Kingdom’s  Cognitive  Cockpit  programme). 

Technological  advances  in  both  Artificial  Intelligence  (AI)  and  the  physiological  monitoring 
of  human  performance  may  allow  higher  levels  of  intelligent  support  to  be  realised.  It  is 
believed  that  in  the  future,  IAH  systems  will  be  considered  more  as  fully  integrated, 
intelligent  systems  that  can  adopt  agent-like  properties,  rather  than  as  conventional  systems 
with  a  discrete  (i.e.,  independent)  automation  control  centre.  Future  IAH  systems  will  be  able 
to  (Eggleston,  1997): 

•  Respond  intelligently  to  operator  commands,  and  provide  pertinent  information  to 
operator  requests; 

•  Provide  knowledge-based  state  assessments; 

•  Provide  execution  assistance  when  authorised; 

•  Engage  in  dialogue  with  the  operator,  either  explicitly  or  implicitly;  and, 
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•  Provide  the  operator  with  a  more  useable  and  non-intrusive  interface  by  managing  the 
presentation  of  information  in  a  manner  appropriate  to  the  content  of  the  mission. 

Furthermore,  IAH  systems  will  be  able  to  provide  support  for  the  basic  functions  of 
assessment,  planning,  co-ordinating  and  acting.  In  these  cases,  an  IAH  system  can  propose  a 
candidate  solution  for  the  human,  or  in  the  extreme,  propose,  select  and  execute  the  solution 
for  the  human.  IAH  systems  will  provide  several  of  the  following  functional  capabilities: 

•  Situation  Assessment.  This  functional  capability  supports  the  organisation  of  large 
amounts  of  dynamic  data  into  concepts  at  varying  levels  of  aggregation  and 
abstraction.  Situation  assessment  includes  task,  system  and  world  models  (refer  to 
Figure  17).  It  provides  the  context  in  which  the  aiding  system  is  operating  and  gives 
this  information  to  other  software  processes  and  to  the  human  user; 

•  Planning.  Based  on  the  situation  determined  by  the  Situation  Assessment,  plans  are 
formulated  by  the  aiding  system.  The  plans  may  cover  different  periods  of  time  at 
different  levels  of  abstraction.  With  information  from  the  user  and  environment, 
aiding  systems  can  independently  formulate  and  propose  plans  to  the  human  users, 
and  can  complete  the  details  of  partial  plans  provided  by  the  operators; 

•  Acting.  An  intelligent  aiding  system  is  not  necessarily  a  passive  system,  but  may  have 
the  capability  to  act  on  behalf  of  its  human  operators.  Given  a  set  of  plans  and  an 
evolving  situation,  the  intelligent  aiding  system  may  issue  commands  directly  to  the 
active  elements  of  the  system,  such  as  sensors,  communications,  propulsion,  flight 
controls  and  secondary  support  systems;  and, 

•  Co-ordination  of  Behaviours.  An  intelligent  aiding  system  must  also  be  able  to  co¬ 
ordinate  its  behaviours  at  four  distinct  levels.  At  the  lowest  level,  the  intelligent 
aiding  system  must  co-ordinate  between  its  own  assessment,  planning  and  executing 
processes  to  produce  coherent  behaviours.  It  must  co-ordinate  with  the  operators  at 
the  planning  level  to  receive  direction  and  provide  useful  recommendations.  It  must 
co-ordinate  with  the  operator  at  the  action  level  so  that  the  operator  and  the  aid  can 
act  in  concert.  The  intelligent  aid  must  co-ordinate  with  the  other  independent 
participants  to  avoid  undesirable  conflicts  and  to  satisfy  the  requirements  of  higher 
level  goals. 

Intelligent  Adaptive  Hybrid  systems  are  reviewed  in  Section  8.4. 

1.5  A  Conceptual  Framework  for  Intelligent  Adaptive  Systems 

Computer  based  systems  assist  the  operator  in  a  multitude  of  mental  and  physical  activities 
such  as,  decision  support  systems,  decision  aids,  automation,  adaptive  automation,  intelligent 
automation,  intelligent  adaptive  interfaces,  expert  systems,  knowledge-based  systems,  data 
fusion,  and  information  fusion.  Computer  based  systems  also  have  varying  degrees  of 
autonomy  (tool,  aid,  associate  autonomous  agent)  and  sophistication  (assistant,  associate  or 
coach).  There  is  a  need  to  develop  a  unified  framework  to  describe  these  conceptual 
approaches  using  consistent  and  unambiguous  terminology. 

The  next  section  re-defines  the  problem  space  by  developing  a  framework  which  encompasses 
both  HF  and  HCI  approaches.  The  framework  defines  IASs  as  multi-dimensional,  continuum- 
based  and  dynamic.  The  dimensions  upon  which  all  IASs,  in  terms  of  their  functionality  (i.e., 
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the  action/use  for  which  the  system  is  designed  to  perform)  and  capability  (i.e.,  the  ability 
necessary  to  perform  a  function),  can  be  placed  are  roles  (i.e.,  what  needs  to  be  done),  agency 
(i.e.,  who  is  doing  what  needs  to  be  done),  and  authority  (i.e.,  who  initiates  or  authorises  what 
needs  to  be  done  and  by  whom). 

1.5.1  Human-Machine  Roles 

There  are  many  roles  that  can  be  shared  between  the  human  operator  and  the  machine.  More 
recent  research  has  included  the  roles  undertaken  by  the  human  and  machine  within 
frameworks  of  the  level  of  automation.  Kaber  and  Endsley  (2004)  determined  that  there  are 
four  roles  that  can  be  shared  between  human  and  machine:  monitoring,  generating,  selecting 
and  implementing.  Parasuraman,  Sheridan  and  Wickens  (2000)  applied  a  more  cognitive 
approach  and  determined  that  there  are  five  roles:  sensory  processing,  perception/working 
memory,  decision  making  and  response  selection.  Neisser  (1976)  formulated  the  concept  of  a 
‘perceptual  cycle’,  where  the  interaction  between  human  and  environment  shapes  the  human’s 
perceptions,  decisions  and  actions.  In  this  view,  cognition  is  a  continuous  cycle  of  perception, 
decision  and  action  where  these  processes  occur  in  parallel  and  with  different  foci.  Each  of 
these  processes  provides  both  cognitive  limitations  and  unique  human  strengths.  Similar 
frameworks  can  be  found  in  Perceptual  Control  Theory  (Powers,  1973;  Hendy  et  ah,  2001) 
and  models  of  Situation  Awareness  in  dynamic  decision  making  (Endsley,  1996). 


Figure  2:  Summary  of  Roles  shared  between  Human  and  Machine.  This  figure  represents 
a  synthesis  of  all  of  the  approaches  cited  above: 

•  Information  Acquisition.  Consists  of  the  roles:  observing,  perceiving,  sensory 
processing,  and  monitoring  (e.g.,  the  machine  obtains  information  from  different 
sources  and  presents  the  information  to  the  operator); 

•  Information  Analysis.  Consists  of  the  roles:  orienting,  assessing,  perception/working 
memory,  and  generating  (e.g.,  the  machine  provides  filtering,  distribution  or 
transformation  of  data,  providing  confidence  estimates  and  integrity  checks,  and 
enabling  operator  requests,  and  the  machine  may  also  manage  how  this  information  is 
presented  to  the  operator); 
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•  Decision  Making.  Consists  of  the  roles:  deciding  and  selecting  (e.g.,  the  machine 
provides  support  to  the  operator’s  decision  making  processes,  either  unsolicited  or  by 
operator  request,  by  narrowing  the  decision  alternatives  or  by  suggesting  a  preferred 
decision  based  on  available  data);  and, 

•  Action.  Consists  of  the  roles:  acting,  responding  to  selections,  and  implementing  (e.g., 
the  machine  executes  actions  or  controls  tasks  with  some  degree  of  autonomy). 


Observe 

Perceive 

Sensory  Processing 
Monitoring 


Orient 

Assess 

Perception  /  Working  Memory 
Generating 


Decide 

Selecting 


Act 

Response  Selection 
Implementing 


Figure  2:  Summary  of  Roles  shared  between  Human  and  Machine. 

1 .5.2  Human-Machine  Agency 

The  function  allocation  between  human  and  machine  (assigning  tasks  to  either  human  or 
machine),  and  the  question  of  “who  is  doing  what  needs  to  be  done?”,  has  generated  much 
research  interest  since  the  conception  of  automation  technologies.  Early  endeavours  were 
largely  based  on  approaches  such  as  the  approach  proposed  by  Fitts  (1951). 

Fitts  created  a  set  of  criteria  where  the  human  can  exceed  machine  capability.  These  criteria 
included: 

•  The  ability  to  detect  small  amounts  of  visual  or  acoustic  energy; 

•  The  ability  to  perceive  patterns  of  light  or  sound; 

•  The  ability  to  improvise  and  use  flexible  procedures; 
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•  The  ability  to  store  very  large  amounts  of  information  for  long  periods  and  to  recall 
relevant  facts  at  the  appropriate  time; 

•  The  ability  to  reason  inductively;  and, 

•  The  ability  to  exercise  judgement. 

Fitts  further  created  a  set  of  criteria  where  machines  can  surpass  human  capability.  These 
criteria  included: 

•  The  ability  to  respond  quickly  to  control  signals  and  to  apply  great  force  smoothly 
and  precisely; 

•  The  ability  to  perform  repetitive  and  routine  tasks; 

•  The  ability  to  store  information  briefly  and  then  to  erase  it  completely; 

•  The  ability  to  reason  deductively  (including  computational  ability);  and, 

•  The  ability  to  handle  highly  complex  operations  (i.e.,  to  perform  many  different 
functions  simultaneously). 

The  sort  of  function  allocation  advocated  by  Fitts  ignores  the  fundamental  question:  how  do 
changes  to  the  allocation  of  tasks  affect  operator  performance?  The  cognitive  abilities  and 
deficiencies  of  an  operator  (such  as  attention,  learning,  boredom,  fatigue,  etc.)  are  of 
paramount  importance  to  the  safe  operation  of  a  system.  While  machines  can  out  perform  an 
operator  in  a  monitoring  task,  humans  excel  in  the  ability  to  adapt  when  situations  change. 

The  issue  is  not  that  Fitt’s  list  is  incorrect;  contemporary  approaches  are  guided  by  more 
recent  knowledge  of  the  strengths  and  limitations  of  human  performance  from  cognitive 
psychology.  The  issue  is  that  this  list  was  often  (erroneously)  taken  to  imply  that  these 
capabilities  were  static  and  separate. 

1 .5.3  Human-Machine  Authority 

Finally,  the  question  of  “who  initiates  or  authorises  what  needs  to  be  done,  and  by  whom?”, 
was  once  thought  of  as  relatively  binary  (i.e.,  the  authority  rests  with  either  the  human  or  the 
machine).  Perhaps  the  most  commonly-cited  taxonomy  of  the  allocation  of  function  between 
human  and  machine  was  produced  by  Sheridan  and  Verplank  (1978)  (see  Table  1 :  Levels  of 
automation. 

The  taxonomy  describes  levels  of  automation  ranging  from  the  human  to  the  machine  being  in 
control.  When  the  human  is  in  control  the  operator  makes  virtually  all  the  decisions  and 
carries  them  out.  When  the  machine  is  in  control  it  decides  whether  a  task  must  be  completed, 
and  only  informs  the  human  if  the  task  is  deemed  appropriate.  There  is  a  clear  dichotomy  in 
this  taxonomy;  the  human  is  in  control  of  the  automation  from  levels  1  through,  5  and  the 
machine  is  in  control  of  the  automation  from  levels  6  through  10. 

As  level  of  automation  increases  from  levels  1  through  3  (i.e.,  decision  making  tool),  levels  4 
through  6  (i.e.,  Operator  Assistant),  levels  7  through  8  (i.e.,  Operator  Associate),  and  levels  9 
through  10  (i.e.,  Autonomous  Agent),  the  amount  of  responsibility  for  higher  level  functions, 
such  as  Decision  Making  and  Action,  also  increases. 
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Table  1:  Levels  of  automation  (Sheridan  and  Verplanck,  1978). 


Level 

Description 

1 

The  machine  offers  no  assistance,  human  must  do  all  the  tasks. 

2 

The  machine  offers  a  complete  set  of  action  alternatives,  and 

3 

narrows  the  selection  down  to  few,  or 

4 

suggests  one,  and 

5 

executes  that  suggestion  if  the  human  approves,  or 

6 

allows  the  human  a  restricted  time  to  veto  before  automation  execution,  or 

7 

executes  automatically,  then  necessarily  informs  the  human,  or 

8 

informs  him  after  the  execution  only  if  he  asks,  or 

9 

informs  him  after  the  execution  if  it,  the  computer,  decides  to. 

10 

The  computer  decides  everything  and  acts  autonomously,  ignoring  the  human. 

Sheridan’s  and  Verplank’s  taxonomy  was  originally  conceived  for  tele-operation  activities 
and  focuses  little  about  the  roles  involved  in  Information  Acquisition  and  Information 
Analysis.  In  response  to  this  limitation,  Endsley  and  Kaber  (1999)  modified  this  taxonomy  to 
include  the  roles  of  monitoring  (i.e.,  Information  Acquisition),  and  generating  (i.e., 
Information  Analysis),  as  well  as  the  roles  of  selecting  (i.e.,  Decision  Making),  and 
implementing  (i.e.,  Action)  (see 

Table  2). 


Table  2:  Levels  of  automation  (Endsley  and  Kaber,  1999). 


Roles 


Level  of  automation 

Monitoring 

Generating 

Selecting 

Implementing 

0) 

Manual  control  (MC) 

Human 

Human 

Human 

Human 

(2) 

Action  support  (AS) 

Hu  man/Compu  ter 

Human 

Human 

Human/Computer 

(3) 

Batch  processing  ( BP) 

Human/Computer 

Human 

Human 

Computer 

(4) 

Shared  control  (SHC) 

Human/Computer 

Human/Computer 

Human 

Human/Computer 

(5) 

Decision  support  (DS) 

Human/Computer 

Human/Computer 

Human 

Computer 

(6) 

Blended  decision  making  (BDM) 

Human/Computer 

Human/Computer 

Human/Computer 

Computer 

(7) 

Rigid  system  (RS) 

Human/Computer 

Computer 

Human 

Computer 

IN, 

Automated  decision  making  (ADM) 

Human/Computer 

Human/Computer 

Computer 

Computer 

(9) 

Supervisory  control  (SC) 

Human/Computer 

Computer 

Computer 

Computer 

(10)  Full  automation  (FA) 

Computer 

Computer 

Computer 

Computer 

Human-Machine  authority  can  be  viewed  as  a  continuum  between  adaptive  systems  (i.e., 
machine-initiated  adaptivity)  on  one  end  of  the  continuum,  and  adaptable  systems  (i.e., 
human-initiated  adaptivity)  on  the  other  end,  with  varying  degrees  of  human  authority  and 
involvement  in  the  middle  (Oppermann  and  Simm,  1994;  see  Figure  3). 
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Figure  3:  Spectrum  of  Adaptivity  (Oppermann  and  Simm,  1994). 

This  classification  of  adaptive  systems  on  basis  of  authority  comprises  five  categories  based 
on  information  the  human  is  given  about  the  systems  status,  and  how  much  control  the 
machine  and  human  have  over  the  initiation  of  the  adaptation.  Oppermann  and  Simm  define 
these  categories  as: 

•  Adaptive.  The  machine  has  total  control  over  adaptation; 

•  System-initiated  adaptivity.  The  machine  will  notify  the  human  of  any  changes  prior 
to  their  execution.  The  operator  still  has  no  control  over  the  choice,  timing  or 
implementation  of  adaptation; 

•  Operator  selected  adaptation.  Using  suggestions  from  the  machine,  the  human  selects 
the  adaptation.  The  machine  still  performs  the  action; 

•  Operator-initiated  adaptability :  The  human  chooses  and  initiates  the  adaptation, 
without  any  suggestions  from  the  machine,  but  the  machine  implements  the  change; 
and, 

•  Adaptable.  The  human  is  in  complete  control  of  adaptation. 

Oppermann  and  Simm’s  spectrum  of  activity  is  a  good  example  of  the  lack  of  integration 
between  the  HF  and  HCI  fields  of  research.  Sheridan  and  Verplanck  were  not  acknowledged 
by  Oppermann  and  Simm  even  though  there  are  obvious  conceptual  similarities  between  their 
taxonomic  classifications. 


1.5.3. 1  Human  versus  Machine  Authority 

There  are  two  main  modes  of  control  over  function  allocation  (Rieger  and  Greenstein,  1982). 
Explicit  allocation  refers  to  situations  where  the  operator  has  allocation  control  over  whether 
tasks  are  to  be  performed  automatically  by  the  machine  or  manually  by  the  operator.  Implicit 
allocation  refers  to  machine  allocation  of  tasks  (Tattersall  and  Morgan,  1996).  When 
comparing  explicit  and  implicit  modes  of  adaptive  automation  research  indicates  that  although 
most  operators  prefer  explicit  control,  implicit  adaptive  automation  is  superior  in  terms  of 
overall  system  performance  (Greenstein  Arnaut,  and  Revesman,  1986;  Lemoine  Crevits, 
Debernard,  and  Millot,  1995).  Although  implicit  adaptive  automation  affords  lower  levels  of 
operator  workload,  a  trade-off  has  to  be  made;  there  is  an  increased  risk  of  operator  out-of- 
the-loop  problems  (see  Section  10.1.1). 
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Many  approaches  (Cook,  Woods,  McColligan,  and  Howie,  1990;  Endsley,  and  Kiris,  1995; 
Lee  and  Moray,  1992)  to  adaptive  automation  assume  that  operator  control  over  function 
allocation  is  preferable  to  the  machine  having  control;  or  at  the  least  consent  should  be 
mandatory.  This  assumption  reflects  the  error-critical  nature  of  the  domains  that  are 
researched  (e.g.,  aviation,  war-fighting,  process  control).  Harris,  Goernert,  Hancock  and 
Arthur  (1994)  looked  at  the  comparative  effectiveness  of  machine-initiated  automation  and 
operator-initiated  automation  during  anticipated  and  unanticipated  increases  in  task  load. 
When  participants  received  written  warnings  that  workload  increases  were  likely  to  occur, 
performance  during  the  operator  and  machine-initiated  automation  did  not  differ.  When  there 
was  no  warning  before  workload  increase,  resource  management  error  was  greater  during 
periods  of  operator-initiated  automation.  The  results  suggest  that  machine-initiated 
automation  is  most  beneficial  when  rapid  workload  increase  occurs  without  warning,  and  that 
when  operator  initiation  is  necessary,  responses  to  rapid  task  load  increases  improve  when 
warnings  are  provided. 


1 .5.4  R-A-A  Framework  for  Intelligent  Adaptive  Systems 

It  is  noted  that  all  intelligent  adaptive  systems  cited  in  this  review  share  the  same  three 
attributes:  role ,  agent ,  and  authority.  The  R-A-A  (Roles- Agent- Authority)  framework  was 
then  developed  here  to  classify  all  the  systems  along  these  three  discrete  dimensions  as 
illustrated  in  Figure  4.  These  three  dimensions  are: 

1 .  Role.  What  are  the  tasks/activities  that  need  to  be  done:  information  acquisition, 
information  analysis,  decision  making,  and  action; 

2.  Agent.  Who  is  performing  the  role:  the  human,  machine  or  both;  and, 

3.  Authority.  Who  authorises  or  initiates  the  Agent  performing  the  Role:  human, 
machine  or  both. 

For  an  intelligent  adaptive  system,  all  tasks  the  system  performs  can  be  divided  into  four 
categories:  information  acquisition,  information  analysis,  decision  making,  and  action.  The 
system  needs  to  be  designed  to  have  relevant  functions  to  conduct  these  tasks.  An  agent  needs 
be  assigned  to  play  the  role  to  perform  these  tasks.  The  decision  of  which  agent  (either  a 
human  or  a  machine  or  both)  to  initiate  the  tasks  needs  to  be  made  by  either  a  human  or  a 
machine  or  both. 
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AUTHORITY 


Figure  4:  The  Cognitive  Cockpit  Intelligent  Adaptive  System  classified  under  the  dimensions 
of  Role,  Agent  and  Authority. 


The  Cognitive  Cockpit  programme  (Figure  4)  is  a  typical  example  to  illustrate  how  this 
framework  will  be  used  to  classify  all  intelligent  adaptive  systems  reviewed  in  this  work. 
Allocation  of  function  between  the  operator,  or  pilot,  and  the  machine  is  very  flexible  in  the 
Cognitive  Cockpit:  both  pilot  and/or  the  machine  can  take  on  the  roles  of  information 
acquisition,  information  analysis,  decision  making  and  action.  The  machine  is  only  able  to 
perform  these  roles  under  the  consent  of  the  pilot;  with  the  exception  of  some  actions  that 
might  take  place  if  the  pilot  is  incapacitated  (e.g.,  machine-initiated  control  of  the  aircraft  in 
the  event  the  pilot  loses  consciousness  in  a  high  gravity  turn).  It  is  important  to  note  that  the 
RAA  framework  represents  a  full-range  of  system  tasks. 
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1.6  Literature  Review  Objectives 

Functional  architectures  commonly  exhibit  the  following  attributes: 

•  The  ability  to  predict  operator  expectations,  intentions  and  actions  based  on  detailed 
embedded  knowledge  of  mission  plans,  goals,  activities  and  alternatives  and  of  the 
environment; 

•  A  model  of  human  decision  making  and  control  abilities,  and  of  communication  with 
other  human  and  non-human  agents; 

•  The  ability  to  monitor  operator  performance  and  workload  through  behavioural  and 
physiological  indices;  and, 

•  The  ability  to  adapt  system  activities  to  the  external  situation  and  the  changing 
abilities  as  well  as  the  limitations  of  the  operator.  This  adaptation  could  involve  the 
fusion  or  filtering  of  information  for  displays,  the  management  of  workload,  or  the 
presentation  of  information  tailored  to  the  operator’s  cognitive  style. 

These  architectures  enable  intelligent  adaptive  systems  to  provide  flexible,  ‘intelligent’ 

assistance  to  an  operator,  in  the  context  of  both  the  operator’s  needs  and  the  external 

situation. 

The  objectives  of  the  literature  research  were  to: 

1 .  Review  theoretical  frameworks,  and  compare  them  to  design  concepts  developed  in 
the  DRDC  UAV  Interface  Design  project; 

2.  Review  analytical  approaches  in  order  to  capture  the  requirements  of  the  OMI  display 
as  well  as  communication  and  control,  and  the  functional  decomposition  of  the 
domain  envisaged  for  the  IAI.  In  addition,  identify  means  to  capture  more  detailed 
knowledge  from  Subject  Matter  Experts  (SMEs)  for  embedding  in  a  Knowledge 
Based  System  (KBS).  This  process  provides  information  about  the  states  of  the 
platform  within  a  mission  context  and  provides  a  basis  for  the  adaptation  of  the 
interface  to  support  the  operator; 

3.  Review  recent  approaches  to  understanding  and  aiding  human  interaction  in  real- 
world  systems  from  a  multi-agent  perspective; 

4.  Review  techniques  for  the  analysis  of  the  psychological,  physiological  and 
behavioural  states  of  the  operator  in  order  to  provide  information  about  the  objective 
and  subjective  state  of  the  operator  within  a  mission  context.  As  with  knowledge  of 
the  external  context,  information  about  the  internal  (i.e.,  operator)  context  provides 
the  basis  for  an  intelligent  adaptation  of  the  interface  to  support  the  operator  to 
achieve  system  goals; 

5.  Attempt  to  consolidate  the  HF  and  HCI  approaches  to  IASs  by  developing  consistent 
terminology  that  maps  to  both  approaches  by  focusing  on  system  functionality  and 
capability;  and, 

6.  Develop  guidance  for  developers  to  assist  in  the  successful  design,  development  and 
implementation  of  IASs. 

Each  objective  was  achieved. 
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2  Method 


This  section  outlines  the  methods  used  to  review,  and  select  the  literature  on  IASs. 

2.1  Approach 

The  approach  to  the  literature  review  had  three  steps: 

•  Literature  search.  Following  the  development  of  appropriate  search  criteria  approved 
by  the  Scientific  Authority  (SA)  (e.g.,  keywords,  authors,  organizations),  the  CAE 
Professional  Services  (Canada)  (CAE  PS)  team  searched  scientific,  defence  (e.g., 
Canada,  United  Kingdom,  United  States  defence  and  NATO  reports),  government 
(e.g.,  DRDC  Toronto  reports  and  other  documentation  provided  by  the  SA)  and 
internet-based  sources  for  literature  pertaining  to  intelligent  adaptive  systems; 

•  Reduce  and  collate  literature.  In  collaboration  with  the  SA,  the  CAE  PS  team 
developed  an  Excel  spreadsheet  database  to  classify  references  by  Level  of 
Experimentation,  Peer  Review,  Domain  Relevance  and  Literature  Review  Area.  The 
results  of  the  literature  search  were  collated  and  reduced  according  to  appropriate 
selection  criteria.  Each  article  was  appraised  for:  the  degree  of  peer  review  (e.g., 
technical  report,  conference  proceedings,  journal  article),  degree  of  experimentation 
involved  (e.g.,  conceptual  study  involving  no  experimentation,  laboratory-based 
experimentation,  field-based  studies),  and  proximity  to  military  domains. 
Recommendations  or  guidelines  developed  from  sources  involving  experimental 
studies  within  the  military  domain  that  have  been  subject  to  critical  peer  review  were 
given  more  prominence  in  the  report  than  those  that  have  been  developed  from  more 
generic,  conceptual  sources,  or  that  were  subject  to  little  or  no  peer  review.  All 
references  were  earmarked  for  inclusion  into  or  exclusion  from  the  final  review,  and 
classified  according  to  area  (i.e.,  conceptual  frameworks,  analytical  techniques, 
design  principles,  and  physiological/behaviour-based  adaptation); 

•  Development  of  reporting  structure.  From  the  collated  literature,  a  structure  was 
developed  for  reporting  findings  in  conjunction  with  the  SA;  and, 

2.2  Structure 

The  objectives  described  in  Section  7.1  were  used  to  structure  the  literature  review.  The 

literature  review  was  looked  at  the  following: 

1 .  Conceptual  Frameworks.  This  is  a  review  of  theoretical  frameworks,  such  as  those 
adopted  by  the  Cognitive  Cockpit,  Pilot’s  Associate  programmes,  and  the  DRDC 
UAV  Interface  Design  project.  The  literature  research  reviewed,  selected  and 
described  conceptual  frameworks  for  designing  IASs,  including,  but  not  limited  to, 
those  described  above.  In  addition,  the  review  also  highlighted  important  similarities 
and  differences,  and  advantages  and  disadvantages,  between  the  theoretical 
approaches; 
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2.  Analytical  Techniques.  This  is  a  review  of  analytical  techniques  that  capture  the 
requirements  and  analyze  the  OMI  display,  communication  and  control  as  well  as  the 
functional  decomposition  of  the  domain  imagined  for  the  intelligent  adaptive  system. 
The  review  of  analytical  techniques  provided  a  means  of  capturing  more  detailed 
knowledge  from  Subject  Matter  Experts  for  embedding  in  a  Knowledge  Based 
System.  These  processes  provide  information  about  the  objective  states  of  the 
platform  within  a  mission  context.  They  also  provide  a  basis  for  the  adaptation  of  the 
automation  and/or  interface  to  intelligently  support  the  operator.  The  review  also 
highlighted  important  advantages  and  disadvantages  between  the  analytical 
approaches; 

3.  Agent-based  Design  Principles.  This  is  a  review  of  approaches  to  understanding  and 
aiding  human  interaction  in  real-world  systems  from  a  multi-agent  perspective.  The 
literature  research  reviewed,  selected  and  described  issues  relevant  to  the 
understanding  and  interaction  between  human  and  machine  agents  in  the  design  of 
IASs  (e.g.,  team  work,  organisation);  and, 

4.  Operator-state  Monitoring  Approaches.  This  is  a  review  of  techniques  for  the 
analysis  of  the  psychological,  physiological  and  behavioural  states  of  an  operator  in 
order  to  provide  information  about  the  objective  and  subjective  state  of  an  operator 
within  a  mission  context.  As  with  knowledge  of  the  external  context,  information 
about  the  internal  (i.e.,  operator)  context  provides  the  basis  for  an  intelligent 
adaptation  of  the  automation  and/or  interface  to  support  the  operator  to  achieve 
system  goals.  The  literature  research  reviewed  technologies  for  designing  behaviour- 
based  and  physiological-based  interface  systems,  and  compared  differences  between 
behaviour-based  and  physiological-based  techniques  and  also  identified  the  benefits 
of  combining  the  two  techniques. 

The  literature  review  is  summarized  in  Figure  4.  The  three  sections  of  the  literature  review 
relate  to  a  typical  human-machine  system  development  structure:  conceive,  analyze,  design 
and  implement.  The  content  of  each  of  the  four  sections  is  also  outlined. 
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*  Conceptual  frameworks 

*  Project- related  frameworks 


•Task/Goal  analysis  techniques 

*  Knowledge  capture  techniques 

*  Automation  analysis  techniques 


*  General  Agent-based  design  principles 

*  Operator-Agent  interaction  principles 

*  Operator  considerations 


*  Psychophysiological-based  implementation 

*  Behaviour-based  implementation 


Figure  4:  Overview  of  literature  review  structure. 


2.3  Summary  of  Articles  Reviewed 

The  following  section  outlines  the  number  and  type  of  articles  reviewed.  Table  3  describes  the 
number  of  articles  allocated  to  each  of  the  four  literature  sections:  conceptual  frameworks; 
analytical  techniques;  agent-based  design  principles;  and,  closed-loop  adaptation 
implementation. 


Table  3:  Number  of  references  used  in  the  literature  review  grouped  by  topic. 


Literature  Review  Topic  Area 

Conceptual 

Frameworks 

Analytical 

Techniques 

Agent-based 

Principles 

Closed-loop 

Adaptation 

Total 

References 

68 

32 

113 

24 

36 


Table  4  describes  the  number  of  articles  classified  in  terms  of  the  level  of  experimentation 
involved,  the  degree  of  peer  review  as  well  as  the  proximity  and  relevance  to  military 
domains.  The  level  of  experimentation  had  four  categories:  conceptual,  single  lab  evaluation, 
single  sim/field  evaluation.  ‘None’,  ‘conference’  and  ‘journal’  were  three  categories  of  peer 
review.  Under  domain  relevance  was  ‘basic’,  ‘business’,  ‘industrial’  and  ‘military’. 


Table  4:  Number  of  references  grouped  by  level  of  experimentation,  peer  review  and 
domain  relevance. 


Level  of  Experimentation 

Peer  Review 

Domain  Relevance 

Conceptual 

Single  Lab 
Evaluation 

Single 

Sim/Field 

Evaluation 

Multiple 

Evaluation 

None 

Conference 

Journal 

Basic 

Business 

Industrial 

Military 

Total 

References 

63 

27 

39 

29 

47 

88 

24 

36 

14 

20 

90 

The  statistics  show  that: 

1 .  A  large  number  of  articles  have  been  written  in  all  four  topic  areas; 

2.  The  articles  are  mostly  conceptual  or  single  laboratory-based  studies  (57%); 

3.  The  articles  are  mostly  subject  to  little  peer  review  (85%);  and, 

4.  A  significant  proportion  of  the  articles  are  from  the  military  domain  (28%). 

2.4  Critique  of  Literature 

The  literature  reviewed  in  this  report  was  evaluated  as  followed: 

•  Each  article  is  appraised  in  terms  of  its  domain  relevance  (i.e.,  proximity  to  the 
military  domain),  and  scientific  impact  (i.e.,  degree  of  peer-review  and  level  of 
experimentation);  and, 

•  A  summary  table  following  every  section  of  the  report  has  been  developed  to  describe 
the  advantages  and  disadvantages  of  the  approaches,  methodologies  and  frameworks. 
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2.5  Example  of  Reference  Format 

Each  reference  was  formatted  according  to  the  format  presented  in  Table  5  (also  see 
accompanying  legend).  The  intention  was  to  impose  a  consistent  and  logical  format  to  assist 
the  reader  in  extracting  the  important  information  and  guideline(s)  quickly. 


Table  5:  Example  of  reference  format. 


®  B 

Reference:  ^ 


Taylor,  R.M.  (2001 ).  Technologies  for  supporting  human  cognitive  control.  In  Proceedings  of  the 
RTO  HFM  Specialists’  Meeting  on  Human  Factors  in  the  21st  Century,  Paris,  France,  11-13  June 
2001. 


Overview:  (3) 


Details  proof-of-concept  demonstration  of  the  Cognitive 
Cockpit  research  program  that  seeks  to  couple  on- 
monitoring  of  pilot  functional  state  assessment,  environment 
and  mission  plan.  Framework  used  to  base  KBS  (roles)  and 
adaptation  (automation)  was  feed-forward  (operator  and 
system)  and  feed  backward  (system)  control  tasks. 

CommonKADS  knowledge  engineering  methodology  led  to 
development  of  several  knowledge-level  models: 
organisational,  task,  agent,  knowledge,  communication  and 
design  models. 

The  Cognitive  Cockpit  is  a  multi-agent  system: 

Cognition  Monitor  (COGMON):  monitors  pilot  functional  state  (level  of  arousal  and  workload). 

Based  on  cognitive  model. 

Situation  Assessor  (SASS):  monitors  environmental  and  aircraft  state  and  recommends  actions. 
Based  on  organization,  task  and  knowledge  models.  Provides  info  about  aircraft,  within  mission 
context  and  supports  decision  process. 

Tasking  Interface  Manager  (TIM):  implements  adaptation  based  on  COGMON  and  SASS  (e.g., 
maximum  goodness  of  fit  between  aircraft  status,  pilot  state  and  tactical  assessments). 

Pilot  Authorizing  and  Control  Tasks  (PACT):  operator  initiative  decision  aiding  for  implementing 
automation.  Based  on  roles/stages  of  information  processing. 

Dynamic  adaptive  interface :  automatically  assigns  roles  to  system  or  operator  according  to 
operator  agreed,  context-sensitive  adaptive  rules. 


© 


ROLE 

AGENT 

AUTHORITY 

ACQUIRE 

a  © 

© 

ANALYZE/ 

PRESENT 

B  © 

© 

DECIDE 

B  © 

© 

ACT 

a  © 

a© 

38 


Conclusions  for  IASs:  q 

1 .  Need  to  be  based  on  User  Centred  Design  (UCD). 

2.  CommonKADS  methodology  and  PC  PAC  software  toolkit  for  knowledge  engineering 
useful  for  implementing  KBS. 

3.  Timing  is  critical  for  effective  contextual  KBS  advice. 

4.  Refer  to  requirements  analysis  within  Cognitive  Cockpit  project  for  specific  analyses  used 
for  development  of  each  agent. 


©  Ratings.  An  iconic  representation  of  the  impact  of  the  article.  Impact  is  defined  as: 


Level  of  experimentation:  Captures  the  weight  of  argument  behind 
guidelines  identified  from  the  article.  This  is  expressed  in  terms  of 
the  following  colour-coding:  red  =  conceptual,  non-experimental 
study;  yellow  =  single  laboratory  experiment  or  single  simulator/field 
experiment;  and,  green  =  multiple  experimental  studies. 


Degree  of  peer  review:  Captures  the  confidence  in  which  the  weight 
ascribed  to  an  article  using  the  criteria  above.  This  is  expressed  in 
terms  of  the  following  colour-coding:  red  =  no  peer-review;  yellow  = 
cursary  peer-review  typical  of  most  conference  proceedings;  and, 
green  =  intense,  critical  peer-review  typical  of  most  journals. 


Proximity  to  military  domain:  Captures  the  proximity  of  an  article  to 
the  military  domain.  This  is  expressed  in  terms  of  the  following 
colour-coding:  red  =  no  specific  target  domain;  yellow  =  industrial  or 
business  target  domains;  and,  green  =  military  target  domain. 


©  Reference.  Full  reference  of  article. 


©  Overview.  Summary  of  the  main  conceptual  or  empirical  points  of  the  reference. 


©  System  Classification  (if  applicable).  Tabular  representation  of  the  type  of  system 
discussed  in  the  article  in  terms  of  Role  (i.e.,  what  needs  to  be  done),  Agent  (i.e.,  who 
is  doing  what  needs  to  be  done)  and  Authority  (i.e.,  who  authorises  and/or  initiates 
what  needs  to  be  done  and  by  whom).  This  classification  is  described  in  more  detail  in 
Section  6.4.  The  human  operator  is  depicted  by  the  ©  icon,  and  the  machine  is 
depicted  by  the  Q  icon. 

©  Conclusions  for  IASs.  A  list  of  design  guidelines  or  recommendations  taken  from 
the  article  that  is  relevant  to  the  development  of  an  IAS. 
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3  Conceptual  Frameworks  for  Intelligent  Adaptive 
Systems 


3.1  Introduction 

The  origins  of  intelligent  adaptive  systems  are  in  the  early  stages  of  development  of  the  crew 
adaptive  cockpit  (Reising,  1979).  The  development  of  intelligent  aiding  from  that  point 
forward  is  tracked  in  the  series  of  USAF/RAF  conferences  on  teamwork  with  the  Electronic 
Crewmember  (Taylor  and  Reising,  1999).  Intelligent  Aiding  systems  previously  attempted, 
and  those  currently  under  construction  operate  at  the  level  of  Assistant  (e.g.,  Germany 
CASSY/CAMMA,  France  Co-pilote  Electronique)  and  Associate  (e.g.,  USAF  Pilots’ 
Associate  and  US  Army  Rotorcraft  Pilots’  Associate  Programmes).  However,  technological 
advances  in  both  Artificial  Intelligence  and  physiological  monitoring  of  human  performance 
have  the  potential  for  allowing  higher  levels  of  Intelligent  Aiding  to  be  realised,  such  as 
providing  an  operator  with  a  more  useable  and  non-intrusive  interface  by  managing  the 
presentation  of  information  in  a  manner  appropriate  to  the  mission  content  (i.e.,  intelligent 
adaptive  interfaces). 

Conceptual  frameworks  are  required  to  enable  such  systems  to  provide  ‘intelligent’  assistance 
to  an  operator  in  the  context  of  the  operator’s  needs  and  the  external  situation.  The 
requirement  to  provide  support  in  the  appropriate  internal  and  external  ‘context’  is  then 
implemented  through  a  functional  architecture  reflecting  the  attributes  of  the  conceptual 
framework  (Taylor  and  Reising,  1999): 

•  A  model  of  human  decision  making  and  control  abilities; 

•  The  ability  to  monitor  operator  performance  and  workload  through  behavioural  and 
physiological  indices;  and, 

•  The  ability  to  predict  operator  expectations  and  intentions  with  reference  to  embedded 
knowledge  of  mission  plans  and  goals. 

Sections  8.2  through  8.4  review  a  number  of  conceptual  frameworks  for  designing  intelligent 
adaptive  systems  including,  but  not  limited  to  those  frameworks  identified  above.  The  review 
also  highlights  important  similarities  and  differences,  and  advantages  and  disadvantages, 
between  the  conceptual  approaches.  The  sections  are  structured  according  to  Intelligent 
Adaptive  Interfaces,  Intelligent  Adaptive  Automation,  and  Intelligent  Adaptive  Hybrid 
systems.  As  discussed  in  Section  6.1,  IAHs  reflect  the  utilisation  of  both  adaptable  automation 
and  an  adaptable  interface  within  the  same  system  (Figure  1). 
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3.2  Intelligent  Adaptive  Interface  (IAI)  Frameworks 

This  section  reviews  the  following  IAI  frameworks: 

•  Situation  Awareness  Assistant  (SAWA); 

•  Stock  Trader; 

•  Personal  Web  Searcher; 

•  Decision-Theoretic  InterAction  Manager  for  Discourse  (DIAManD); 

•  Work-centered  Decision  Support  (WCSS); 

•  Adaptive  Icon  Toolbar;  and, 

•  ConCall  System. 


3.2.1  Situation  Awareness  Assistant  (United  States) 


Reference: 


Matheus,  C.J.,  Kokar,  M.M.,  Baclawski,  K.,  Letkowski,  J.J.,  Call,  C.,  Hinman,  M.,  Salerno,  J.,  and 
Boulware,  D.  (2005).  Lessons  learned  from  developing  SAWA:  A  Situation  Awareness  Assistant. 
Technical  report  Air  Force  Research  Laboratory,  Rome,  NY. 


Overview: 

This  paper  details  the  Situation  Awareness  Assistant  (SAWA) 
project  and  various  lessons  learned  during  its  development; 
including  the  pros  and  cons  of  leveraging  semantic  web 
technologies,  the  handling  of  time-varying  attributes  and  the 
processing  of  uncertainty. 

SAWA:  The  authors  view  situation  awareness  as  a  fusion 
problem.  Therefore,  the  SAWA  project  developed  specific 
domain  knowledge  database  offline.  This  database  can  then 
be  applied  in  real-time  to  fuse  and  analyse  data. 

SAWA  Process: 

1 .  Domain  knowledge  is  captured  in  SAWA  using  formal  ontologies.  Formal  ontologies  can 
provide  a  flexible  query  and  monitoring  language  that  can  be  used  to  request  information  to 
increase  situation  awareness.  These  queries  can  include  information  about  the  current 
situation,  predicted  situations  and  request  notifications  of  current  or  potential  future  emergency 
conditions. 

2.  Semantic  Web  Ontology  Language  (OWL),  is  used  to  define  ontologies  which  provide  a  formal 
set  of  semantics.  Data  and  knowledge  representation  use  these  semantics  as  a  basis  within 
the  SA  system. 

3.  Semantic  Web  Rule  Language  (SWRL)  is  used  on  top  of  OWL  to  define  portions  of  the  domain 
knowledge  using  rules. 
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4.  Situation  Awareness  Core  Ontology  was  used  to  develop  specific  domain  knowledge 
ontologies  and  rule  sets. 

The  SAWA  High-level  Architecture  has  two  aspects:  a  set  of  offline  tools  for  Knowledge 
Management  and  a  Runtime  System  of  components  for  applying  domain  knowledge  to  the 
monitoring  of  evolving  situations. 

The  operator  interacts  with  the  system  through  a  Graphical  User  Interface  (GUI)  by  executing 
queries  and  monitoring  the  current  state  of  events. 

Automation :  Information  is  acquired,  analyzed  and  synthesized  to  assess  current  situations  and 
generate  possible  future  situations  to  support  decision  making. 


Conclusions  for  IASs: 

1 .  The  authors  advocate  that  Semantic  Web  technologies  can  be  used  for  representing  and 
reasoning  about  knowledge  pertinent  to  a  situation’s  domain  but  that  they  are  difficult  to 
implement. 

2.  The  authors  also  claim  that  the  behaviour  of  dynamic  objects  within  its  domain  ontology  should 
be  modelled  in  order  to  provide  up-to-date  situation  awareness  of  all  objects/events  at  all 
times. 

3.  Representation  of  certainty  (or  uncertainty)  should  be  presented  to  ensure  that  the  operator  is 
aware  of  the  system’s  reasoning  processes.  The  authors  resorted  to  using  Bayesian  reasoning 
within  their  inference  engine  to  manage  uncertainty. 


3.2.2  Stock  Trader 


Reference: 


Yoo,  J.,  Gervasio,  M.,  &  Langley,  P.  (2003).  An  adaptive  stock  tracker  for  personalized  trading 
advice.  Proceedings  of  the  International  Conference  on  Intelligent  User  Interfaces,  pp.  197 


Overview: 

The  Stock  Trader  system  investigated  operator  performance. 
The  system  addresses  information  overload  by  tailoring 
recommendations  based  on  an  individual  operator's 
investment  styles.  The  system  utilizes  this  profile  to  rank 
stocks,  and  it  revises  the  profile  based  on  traces  of  operator 
behavior.  The  system  automates  information  acquisition;  it 
encompasses  sensing,  and  registers  input  data. 
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The  system  architecture  is  composed  of  the  following  elements: 

1 .  The  data  processing  unit  converts  raw  input  (i.e.,  current  stock  readings  and  historical  trading 
information)  into  reports  that  contain  buy  and  sell  recommendations  for  the  operator.  It  relies 
on  the  recommendation  module  to  make  appropriate  suggestions  for  each  stock  based  on 
individual  operator  profiles. 

2.  The  user  modeler  which  constructs  these  profiles  is  based  on  operator  responses  to  previous 

recommendations  (implicit). _ 
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3.  The  information  manager  records  traces  of  an  operator’s  interactions  with  the  system  and  also 
maintains  awareness  of  operator  portfolios. 

4.  The  communication  unit  manages  the  information  into  and  out  of  the  server. 

5.  A  client  contains  a  communication  unit  and  a  graphical  user  interface  component. 

Results  from  a  study  conducted  with  novice  stock  traders  indicated  that  as  the  system  learned 
through  interaction  with  the  operator’s  past  behaviour,  the  traders’  acceptance  of 
recommendations  increased.  Furthermore,  as  the  traders’  began  to  better  understand  how  the 
system  operates,  they  also  began  to  accept  more  recommendations. 


Conclusions  for  IASs: 

1 .  An  implicit  user  model  is  an  effective  and  non-obstructive  means  of  constructing  a  user  model. 


3.2.3  Personal  Web  Searcher 


Reference: 


Armentano,  M.,  Godoya.D.  and  Amandi,  A.  (2006).  Personal  assistants:  Direct  manipulation  vs. 
mixed  initiative  interfaces.  International  Journal  of  Human-Computer  Studies  64  (2006)  27-35 


Overview: 

This  paper  explores  new  mixed-initiative  metaphors  to 
enhance  an  operator’s  ability  to  directly  manipulate 
interfaces.  Mixed-initiative  interaction  is  referred  to  as  a 
flexible  interaction  strategy  in  which  agents  are  used  to 
manage  information  overload.  A  study  evaluating  how  the 
interaction  metaphor  can  affect  the  operator  perception  of 
agent  capabilities  is  reported. 

The  mixed-interface  is  the  “PersonalSearcher”,  an  intelligent 
agent  that  builds  an  operator  profile  implicitly  by  observing  operator  behaviour  while  operators  are 
performed  regular  activities  on  the  Web.  An  agent  is  able  to  deduce  the  topics  an  operator  are 
interested  in  to  create  an  operator  profile  by  using  a  content-based  analysis  of  the  information 
extracted  by  observation. 

The  study  compared  two  interfaces:  1)  an  operator  interacts  with  the  interface  directly  and  has  no 
control  over  displayed  suggestions  (automation)  and  2)  an  operator  interacts  with  an  animated 
“agent”  instead  of  the  interface  and  has  control  over  suggestions  (mixed-initiative). 

Results  indicate  that  the  mixed-initiative  interface  increased  situational  awareness  (i.e. ,  operators 
noticed  improvements  in  the  agent  suggestions  over  time),  but  that  participants  were  more  critical 
of  suggestions. 
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Conclusions  for  IASs: 

1 .  Mixed-initiative  interfaces  (e.g.,  direct  interaction  with  an  agent)  can  increase  situational 
awareness  and  develop  a  better  mental  model  of  the  system. 

2.  Designers  must  be  careful  when  designing  mixed-initiative  interfaces  to  ensure  a  proper 
mental  model  of  the  system  is  achieved. 


3.2.4  Decision-Theoretic  InterAction  Manager  for  Discourse 


“*  bqb 

Wolfman,  S.A.,  Lau  Pedro  Domingos,  T  and  Weld,  D.S.  (2001).  Mixed  Initiative  Interfaces  for 
Learning  Tasks:  SMARTedit  Talks  Back.  In  proceedings  of  IUI’01,  January  14-17,  2001,  Santa  Fe, 
New  Mexico,  USA. 


Overview: 

An  interface  for  machine  learning  is  proposed.  The  paper 
describes  a  variety  of  interaction  modes  that  enhance  the 
learning  process  and  presents  a  decision-theoretic  framework, 
called  DIAManD,  for  choosing  the  best  interaction. 

The  authors  propose  that  machine  learning  systems  should 
closely  resemble  human  teacher-student  relationships  and 
follow  the  example  of  the  proactive  yet  considerate  student. 

For  instance,  the  system  should  ask  questions,  propose 
examples  and  solutions,  and  relate  its  level  of  knowledge  when  appropriate  to  make  the  interaction 
more  effective. 

DIAManD  is  a  system  for  selecting  among  various  interaction  modes  using  a  multi-attribute  utility 
function.  The  interaction  modes  provide  a  variety  of  methods  for  an  operator  to  interact  with  the 
system.  The  system  selects  from  a  set  of  interaction  modes  the  mode  it  judges  most  appropriate 
based  on  attribute  vectors.  The  best  of  these  modes  is  presented  to  the  operator  and  controls  the 
next  stage  of  discourse,  updating  the  state  of  the  learner.  The  modes  are  then  rescored  based  on 
the  new  state  of  the  learner. 

The  paper  outlines  and  describes  the  interaction  modes. 


Conclusions  for  IASs: 

1 .  The  authors  advocate  a  mixed-initiative  interface  in  which  the  machine  learner  and  human 
operator  equally  share  responsibility  for  guiding  the  learning  process. 

2.  A  learning  system  should  have  several  modes  of  interaction  with  the  operator  to  acquire  the 
concepts  more  quickly  (e.g.,  through  judicious  choice  of  the  example  to  classify,  as  in  active 
learning)  and  should  allow  the  operator  to  have  more  control  over  the  learning  process.  See 
paper  for  details  on  interaction  modes. 

3.  A  mixed-initiative  framework  (e.g.,  DIAManD),  where  the  learner  and  human  operator  are  each 
participants  in  a  dialogue,  could  improve  the  learner's  hypothesis  with  minimal  effort  on  the 
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part  of  the  operator. 

4.  The  operator  of  a  system  should  be  able  to  override  the  system's  choice  of  interaction  mode 
and  choose  a  mode  that  he/she  prefers. 

5.  To  facilitate  rapid  learning,  the  interface  should  provide  some  mechanism  for  feedback  to  the 
learning  system  on  particularly  poor  interaction  mode  choices  (the  feedback  model  is  further 
described  in  the  article). 

6.  An  attribute  set  must  reflect  the  balance  between  operator  effort  and  the  value  to  the  task  and 
system. 

7.  The  authors  recommend  five  appropriate  but  general  attributes,  each  of  which  should  be  viable 
for  most  learning  system  and  interaction  library  combinations.  The  attributes  (operator  input, 
level  of  continuity,  and  probability  of  correction)  focus  on  operator  effort  and  represent  the 
physical  and  mental  effort  required  from  an  operator.  The  attributes  (task  progress  and  value 
to  the  system)  focus  on  the  achievement  of  an  operator's  objective.  These  measures  reflect  an 
operator’s  typical  objective  of  a  machine  learning  system,  which  is:  complete  the  task  by 
refining  the  hypothesis  of  the  learning  system  until  it  correctly  describes  the  data. 


3.2.5  Work-Centered  Decision  Support 


Reference: 


Young,  M.J.  and  Eggleston,  R.G.  (2002).  Work-Centered  Decision  Support.  In  Proceedings  of  RTO 
Human  Factors  and  Medicine  Panel  (HFM)  Symposium  held  in  Warsaw,  Poland,  7-9  October  2002 


Overview: 

The  Work-Centered  Decision  Support  system  is  a  stand-alone 
interface  client  that  manages  information.  The  system 
employs  intelligent  agents  that  dynamically  plug  into  an 
information  grid  to  find,  fuse,  format,  and  present  information 
to  an  operator  in  a  manner  relevant  to  the  current  context.  The 
system  is  based  on  a  task  model  and  work  domain  ontology 
model.  Cognitive  task  analysis  techniques  were  used  to 
acquire  the  information  to  build  the  task/model. 

The  WCSS  system  is  composed  of  three  layers: 

Acquisition  Agent:  This  agent  contains  knowledge  on  how  to  find  and  retrieve  data.  The  agent’s 
function  is  to  automatically  monitor  and  access  data  sources  for  an  operator  and  notify  other 
agents  when  new  data  has  been  retrieved  or  received  (information  acquisition). 

Analysis  Agents:  These  agents  contain  the  knowledge  required  to  transform  data  into  information 
that  will  support  decision  making.  Their  function  is  to  provide  data  reasoning  and  fuse  data  to 
create  patterns  of  information.  These  agents  use  the  data  collected  by  the  acquisition  agents  to 
continually  appraise  the  situation,  proactively  identifying  possible  problems,  and  dynamically 
generating  a  prioritized  list  of  potential  operator  actions  (information  analysis). 

Presentation  Agent:  This  is  a  communications  and  dialogue  module  that  controls  the  information 
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presented  on  the  screen,  the  response  to  operator  requests,  and  the  provision  of  alerts  to  the 
operator  to  identify  potential  problems  and  opportunities.  This  agent  aggregates  or  disaggregates 
information  based  upon  who  the  operator  is,  and  what  the  operator’s  current  requirements  are.  The 
interfacing  agent  is  an  ecological  interface. 

An  example  of  this  framework  was  applied  to  a  Work-Centered  System  for  Global  Weather 
Management  to  support  weather  forecasting  in  a  military  airlift  service  (Refer  to  the  paper  for 
further  details). 

The  authors  claim  that  the  two  most  important  research  areas  for  the  future  development  of  the 
WCSS  are  the  role  of  agents  in  the  interface,  as  well  as  improving  the  creation  of  analysis  agents 
by  developing  extensions  to  cognitive  task  analysis  techniques. 


Conclusions  for  IASs: 

1 .  A  Work  Domain  ontology  was  useful  to  provide  an  organizing  framework. 

2.  The  reference  to  task  domain  elements  in  on-screen  information  displays  should  be  made 
more  explicit. 

3.  Fusion  of  data  from  multiple  databases  can  be  useful  to  identify  and  complete  work  tasks. 

4.  The  authors  believe  that  the  introduction  of  social  agents  could  potentially  reintroduce 
unnecessary  mental  shifts  in  work,  as  the  operator  shifts  from  focusing  on  the  task  to 
“socializing”  with  the  agent.  The  authors  indicate  that  Milewski  and  Lewis  (1997)  claim  that  the 
use  of  agents  as  the  interface  (as  opposed  to  having  “behind  the  scenes”  agents)  usually 
involves  a  delegation  model,  where  the  operator  delegates  activities  to  the  agents.  The  authors 
claim  that  delegating  activities  to  agents  (or  other  humans)  requires  many  processes;  the 
delegator  must  consider  the  competency  of  the  agent,  and  then  communicate  outcomes  and 
possible  strategies,  and  monitor  the  progress  in  work.  These  activities  require  knowledge  types 
different  from  other  than  domain  task  knowledge.  The  authors  suspect  this  would  introduce 
unnecessary  shifts  in  focus,  away  from  the  task,  as  the  operator  applies  “social  knowledge”  to 
commission  the  agent. 

5.  Task  elicitation  techniques  could  be  used  to  identify  models  of  decision  making  and  cues  used 
by  skilled  (or  expert)  decision-makers.  The  analysis  agents  could  then  be  designed  to  identify 
patterns,  and  have  the  presentation  agents  present  a  rank  ordered  set  of  potential  problem 
situations  to  the  operator.  In  the  current  implementation  of  the  WCSS,  operators  must 
manually  search  for  cues,  which  they  then  integrate  mentally.  Automating  this  process  has  the 
potential  to  greatly  improve  decision-making  quality  and  reduce  human  errors. 

6.  More  research  is  required  to  determine  which  techniques  are  best  suited  for  knowledge 
elicitation  of  schemas,  and  then  to  determine  how  to  best  map  this  knowledge  into  analysis 
agents  to  maximize  work  support. 

7.  The  ecological  interface  provides  an  adequate  means  of  supporting  an  operator’s  work  tasks 
by  providing  contextual  situational  awareness;  the  ecological  interface  presents  an  action  by 
making  readily  apparent  what  action  is  required  and  the  constraints  of  that  action  (in  WCC,  the 
action  is  done  by  the  operator). 


Reference: 


Eggleston,  R.G.,  Roth,  E.M.,  Scott,  R.A.  (2003).  A  Framework  for  Work-Centered  Product 
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Evaluation.  In  proceedings  of  the  47th  Human  Factors  and  Ergonomics  Society  Conference, 
Denver. 


Overview: 

A  comprehensive  work-centred  evaluation  framework  that  assesses  new  technology  for  their  value 
in  supporting  human  performance  is  described.  A  key  feature  of  the  framework  is  that  it 
encompasses:  usability,  usefulness  and  impact.  This  concept  is  illustrated  through  a  work-centred 
support  system  prototype.  The  framework  is  detailed  in  Young,  M.J.  and  Eggleston,  R.G.  (2002). 

In  the  WCSS  prototype,  software  agents  are  designed  as  small,  independent  chunks  of  software 
that  address  tasks  as  separately  controlled  and  modifiable  modules.  This  enables  software 
components  to  be  organized  according  to  functional  elements  of  work  in  a  particular  domain. 

A  detailed  domain  analysis  was  performed  to  map  domain  work  requirements  and  systematically 
allocate  tasks  to  human  and  software  agents. 

Structuring  agents  in  functional  terms  provides  a  concrete  vocabulary  of  concepts  and  metaphors 
that  can  be  shared  among  software  engineers,  cognitive  engineers,  and  operators. 

Two  types  of  interfacing  agents  are  used  in  the  prototype.  The  “visibility”  of  the  agents  is  based  on 
the  task: 

a.  Agents  organized  around  domain  work.  These  include  forecasting  agents,  region  analysis  and 
mission  analysis  agents,  which  are  agents  that  operators  “delegate”  work  to;  they  have  no 
personality. 

b.  Agents  that  operators  can  access  if  needed.  These  include  data  acquisition  agents. 


Conclusions  for  IASs: 

1 .  Cognitive  work  analysis  (CWA)  is  an  effective  means  of  establishing  system  requirements.  The 
authors  advocate  that  sources  of  cognitive  and  collaborative  demands  should  be  analyzed  in 
the  applied  domain  and  involve  close  interaction  among  the  cognitive  engineers,  software 
developers  and  domain  practitioners. 

2.  Automated  agents  should  act  as  ‘team  players’. 

3.  Visibility  of  agents:  Automated  agents  need  to  be  observable  (or  transparent/visible)  so  that 
operators  are  able  to  determine  the  current  state  of  the  automated  agents,  and  understand 
what  the  agents  will  do  next  relative  to  the  state  of  the  task.  The  amount  of  “visibility”  required 
is  questionable  (i.e.,  the  issue  of  trust  and  mistrust  can  occur  or  fully  visible  such  as  the 
Microsoft  “PaperClip”  which  takes  advantage  of  assistant  and  subordinate  metaphors) 

4.  Humans  should  have  control  and  be  able  to  re-direct  the  software  agents  as  task  requirements 
change. 

5.  A  system  needs  to  support  multiple  facets  of  individual  cognitive  and  collaborative  work.  This 
involves  consideration  of  problem-solving/decision-making  aspects  of  work,  activities  involved 
in  creating  work  products,  processes  involved  in  collaborative  work,  and  the  cognitive  effort 
involved  in  tracking  and  managing  multiple  intertwined  work  activities. 

6.  Object-oriented  design  techniques  are  useful  in  facilitating  collaboration  between  operators, 
cognitive  engineers,  and  software  engineers  (although  as  system  complexity  increases,  the 
operator  can  lose  sight  of  the  big  picture). 

7.  Agent-based  architectures  provide  potential  for  operator-accessible  descriptions  of  domain 
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objects,  workflow,  and  large-scale  interactions  between  domain  objects. 


Eggleston,  R.G.  (1992).  Cognitive  interface  considerations  for  intelligent  cockpits.  Proceedings  of 
the  AGARD  conference  on  Combat  automation  for  airborne  weapon  systems:  Mlockan/machine 
interface  trends  and  technologies,  Edinburgh,  UK,  19-22  October  1992. 

Overview: 

This  paper  presents  the  concept  of  an  intelligent  cockpit  as  an  example  of  a  knowledge-based 
aiding  system.  Cognitive  design  requirements  for  aiding  systems  are  presented  along  with 
illustrative  examples. 

Compared  to  conventional  cockpits,  the  authors  claim  that  an  intelligent  cockpit  is  much  more 
flexible  and  adaptive  in  handling  events.  Intelligent  cockpits,  as  opposed  to  conventional  ones, 
have  the  ability  to  consider  a  wide  range  of  data  and  issues  that  allow  it  to  exhibit  adaptive 
behavior. 

Cognitive  design  requirements  include  all  system  factors  that  are  essential  for  the  system  to 
behave  at  a  symbolic  and  abstract  level  of  understanding.  The  major  challenge  is  how  to 
adequately  account  for  and  predict  the  form  of  adaptive  behavior  that  an  operator  will  exhibit  in  a 
given  task. 


Conclusions  for  IASs: 

1 .  Knowledge  of  human  capabilities  and  limitations  are  important  factors  for  the  design  of  an 
intelligent  interface;  these  factors  include  but  are  not  limited  to,  attention,  working  memory,  and 
an  analysis  bias  (reasoning  and  decision  making). 

2.  The  cognitive  architecture  for  an  intelligent  system  should  be  designed  so  that  an  appropriate 
method  of  implementation  of  automation  can  be  chosen  that  is  based  upon  the  situation. 


3.2.6  Adaptive  Icon  Toolbar 

Reference: 


Debevc,  M.,  Meyer,  B.,  Donlagic,  D.,  &  Svecko,  R.,  (1996).  Design  and  evaluation  of  an  adaptive 
icon  toolbar.  User  Modeling  and  User-Adapted  Interaction,  6(1),  pp.  1-21. 
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Overview: 

An  adaptive  icon  toolbar  in  MS  Word  was  evaluated.  The  time  and  errors  taken  to  adapt  the 
toolbar  were  examined  during  two  tasks  (formatting  text  and  tables).  The  operator  is  always  in 
control  of  adapting  the  toolbar  but  the  system  offers  suggestions.  Refer  to  the  paper  for  further 
description  regarding  the  decision-making  algorithm  implemented  in  the  toolbar. 

Adaptive  toolbar.  This  is  an  operator-controlled  self-adaptation  system  that  adapts  the  OMI  by 
observing  operator  actions,  making  proposals  to  adapt  the  icon  bar  to  best  reflect  those  actions, 
and,  if  the  operator  agrees,  automating  the  process  of  customizing  the  toolbar. 


Maintaining  system  awareness.  Any  change  or  proposal  for 
change  in  the  OMI  is  made  visible  to  the  operator  by  playing  a 
tone  and  by  changing  the  background  colour  of  the  toolbar. 
The  toolbar  colour  reverts  back  to  normal  once  the  operator 
accepts  or  rejects  the  change.  This  alert  eeps  the  operator  in 
the  loop  while  not  disrupting  work. 


User  model :  The  user  model  is  based  on  operator  interaction 
with  the  system  (i.e. ,  frequency  of  use  of  commands,  options,  and  macros  not  already  directly 
available  in  the  interface).  The  paper  provides  more  details  on  decision  algorithms  for  adaptation. 

Adaptation  uncertainty.  Uncertainty  about  the  adaptation  is  displayed  to  the  operator  to  identify 
when  and  how  to  adapt  the  interface  (e.g.,  via  size  of  the  icon  -  unused  icons  grow  smaller).  The 
adaptation  is  based  upon  operator’s  previous  actions. 


Conclusions  for  IASs: 

1 .  Adaptation  can  increase  learning  of  system  functionality  and  enhance  performance.  Study 
results  from  the  study  showed  that  the  adaptation  of  the  toolbar  allowed  for  more  efficient 
customization  as  operators  spent  significantly  less  time  adapting  the  toolbar.  Adaptation  also 
introduced  novice  operators  to  toolbar  functionality  and  therefore  increased  learning. 

2.  The  authors  suggest  several  possible  goals  and  benefits  of  adaptive  interfaces  including: 
adaptive  interfaces  should  be  easy,  efficient,  and  effective,  and  should  encourage  faster  and 
simplified  use.  Adaptive  interfaces  should  increase  the  ease  of  working  within  complex 
systems  and  should  present  what  the  operator  wants  to  see. 

3.  The  authors  recommend  that  an  operator  interface  should  consider  operator  increasing 
experience  with  the  system  in  order  to  provide  skill-specific  automation. 

4.  Adaptation  of  the  operator  interface  can  increase  efficiency  of  customization  and  learning  of 
interface  features. 
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3.2.7  Concall  System 


Reference: 


Averman,  C.  (1999).  Using  "Human-in-the-loop"  in  an  adaptive  system:  An  evaluation  study  of  the 
ConCall  system.  Unpublished  master’s  thesis,  Goteborg  University,  Goteborg,  Sweden.  Retrieved 
January  3,  2005  from  http://www.handels.qu.se/epc/archive/00001335 


Overview: 

This  paper  discusses  the  evaluation  of  the  Concall  system 
which  is  used  to  reduce  information  overload  by  sorting  and 
filtering  call  for  papers  and  participation  for  conferences.  It  is 
considered  a  mixed  interface;  operators  provide  an  explicit 
user  model  and  the  interface  then  produces  recommendations 
based  on  the  user  model.  Results  indicate  that  the  system 
provided  too  many  recommendations  and  therefore,  operators 
stopped  using  the  system  out  of  frustration.  Operators  also 
wanted  an  “UNDO”  function  and  wanted  to  be  less  disturbed. 

Since  the  system  lacked  a  proper  mental  model,  operators’  were  found  to  request  a  function  that 
would  undo  a  previous  adaptive  change  that  the  system  has  determined  would  be  the  right  choice 
for  the  operator. 
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Conclusions  for  IASs: 

1 .  An  explicitly  elicited  user  model  may  not  be  the  appropriate  means  to  acquire  a  user  model. 

2.  The  operator  must  be  fully  understood  in  order  to  develop  a  user  model  that  will  provide 
efficient  and  usable  adaptivity. 


Section  Summary 

The  frameworks  reviewed  emphasize  the  importance  of  user  models  and  agent  use  with  IAIs. 
These  frameworks  have  been  applied  to  address  the  issues  of  automation  and  human  control 
of  interfaces.  The  most  important  ability  of  an  IAI  is  to  understand  the  user  and  to 
communicate  with  him/her.  A  system  needs  to  have  the  ability  to  intelligently  appraise  a 
situation  and  to  adapt  to  the  changing  needs  of  the  user  and  the  situation.  A  multi-agent 
architecture  is  needed  to  integrate  all  the  agents  working  collaboratively  in  a  system  to 
interact  with  the  user.  To  achieve  these  goals,  a  unified  framework  is  needed  and  the 
associated  analytical  methodologies  should  be  categorized  to  address  different  task  domains. 

3.3  Intelligent  Adaptive  Automation  Frameworks 

This  section  reviews  the  following  IAA  frameworks: 

•  Crew  Assistant  Military  Aircraft  (CAMA); 

•  Co-Pilote  Electronique; 
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•  Delegated  Systems  (Playbook); 

•  Intelligent  Classroom;  and, 

•  Lookout. 

3.3.1  Crew  Assistant  Military  Aircraft  (Germany) 

The  Crew  Assistant  Military  Aircraft  is  a  knowledge-based  system  that  was  developed  by  the 
University  of  the  German  Armed  Forces  (Munich),  DASA,  and  DLR,  for  improving  pilots’ 
situation  awareness  in  air  transport.  CAMA  assists  the  crew  in  planning  and  decision  making 
tasks  through  all  flight  phases. 

CAMA  consists  of  an  “electronic  crew  member”  which  gathers  information  from  the  crew 
through  monitoring  control  actions  and  through  image  processing  of  an  inside  cockpit  camera. 
The  internal  and  external  data  sources  are  connected  to  the  system  by  appropriate  sensors  or 
communication  media.  The  Central  Situation  Representation  is  a  dynamic  object-oriented 
representation  of  relevant  data.  This  representation  contains  all  situation  related  (dynamic) 
and  domain  related  (static)  knowledge.  The  Crew  Interface  is  the  audio-visual  communication 
layer  between  CAMA  and  the  crew.  The  interface  selects  and  co-ordinates  information  to  be 
shown  on  a  2D  map  display  or  to  be  issued  via  a  speech  synthesiser.  The  latter  provides 
system  control  through  speech  recognition.  The  Planning  Layer  generates  a  complete  flight 
mission  plan.  In  the  Situation  Interpretation  Layer  this  flight  plan  is  used  as  a  reference  for  the 
crew  model.  Here,  the  expected  crew  actions  are  elaborated  and  aspects  of  the  external 
situation,  such  as  tactical  elements  and  terrain,  are  evaluated.  The  modules  in  the  Situation 
Assessment  layer  are  intended  to  detect  conflicts  in  the  expected  succession  of  the  flight  and 
to  recognise  the  crew’s  intent  and  error.  In  the  case  of  a  pilot  error,  a  warning  or  hint  is  given 
to  the  crew  to  correct  the  error.  In  order  to  cope  with  the  temporary  discrepancy  of  crew 
intent,  CAMA  attempts  to  extract  the  intention,  modify  the  flight  plan  accordingly,  and 
elaborate  the  consistent  expected  behaviour  again.  The  structure  of  CAMA  mirrors  the 
general  design  philosophy  not  to  solely  allocate  functions  to  the  machine  side  or  the  crew,  but 
to  both  in  parallel. 

In  order  to  incorporate  tactical  elements  into  CAMA’s  planning  and  decision  making,  the 
Tactical  Situation  Interpreter  (TSI)  and  Low- Altitude  Flight  Planner  (LAP)  are  integrated  in 
the  Situation  Interpretation  Layer: 

•  Tactical  Situation  Interpreter.  The  TSI  is  a  knowledge-based  module  in  the  context  of 
the  CAMA  Situation  Interpretation  Layer.  The  TSI  contributes  two  main  functional 
aspects:  the  computation  of  the  Threat  Map  based  upon  a  list  of  tactical  elements  such 
as  surface-to-air  missiles  (SAMs)  and  other  aircraft,  decision  aiding  functions  such  as 
distance  to  threat  calculations  in  order  to  transform  sensor  data  into  an  intuitive 
format  with  respect  to  the  human  information  processing  capabilities  of  the  crew.  The 
calculations  are  based  upon  tactical  elements  which  are  either  static  and  known,  or 
provided  by  external  communication  modules  which  are  fed  into  the  Central  Situation 
Representation;  and, 

•  Low  Altitude  Planner.  The  LAP  calculates  a  low  altitude  trajectory  through  the 
operation  area  based  upon  digital  terrain  data,  the  TSI’s  threat  map,  and  on  the  basis 
of  given  plan  parameters,  such  as  waypoints.  The  process  of  low-level  flight  plan 
generation  consists  of  two  principle  steps:  the  evaluation  of  the  tactical  situation, 
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which  results  in  the  calculation  of  the  Threat  Map,  and  the  low-level  flight  trajectory 
optimisation.  The  trajectory  optimisation  considers  the  terrain  elevation  structure,  the 
local  threat  value  from  the  Threat  Map,  and  constraints  resulting  from  the  general 
mission  plan.  The  result  is  an  optimal  trajectory,  minimising  a  cost  function  of 
weighted  terrain  elevation  data  and  local  threat  values  integrated  over  the  complete 
flight  path.  The  data  are  visualised  in  a  2D  map  display. 


Reference: 


Schulte,  A.  and  Klockner,  W.  (1998).  Crew  Assistant  for  tactical  flight  missions  in  simulator  and 
flight  trials.  NATO  systems  concepts  and  integration  panel  symposium:  The  application  of 
information  technology  (computer  science)  in  mission  systems.  Monterey,  California,  USA.  20-22 
April  1998. 


Overview: 

The  Crew  Assistant  Military  Aircraft  project,  a  result  of  CASSY 
project  efforts,  is  described.  This  is  a  tactical  mission 
management  system  that  assists  in  situation  assessment  and 
fight  planning. 

The  CAMA  system  was  experimentally  evaluated  through  a 
series  of  simulator  flights  with  operational  personnel  (e.g., 
flight  and  simulator  trials).  Results  showed  high  operator 
acceptance  and  a  well-developed  mental  model  of  the  system  following  training  and  repeated 
interactions  with  the  system.  Situation  awareness  concerning  the  tactical  situation  elements 
remained  unchanged. 
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Conclusions  for  IASs: 

1 .  Comprehensive  knowledge  bases  are  necessary  for  the  implementation  of  knowledge-based 
systems. 

2.  Task-based  approach.  Functions  derived  from  tasks  relating  to  tactical  low-level  flight  missions 
proved  to  be  a  successful  in  providing  situation  assessment  and  mission  planning. 


Reference: 


Brugger,  E.  and  Hertweck,  H.  (1995).  CAMA:  Some  aspects  of  a  military  crew  assistant  system. 
Proceedings  of  the  3rd  international  workshop  on  human-computer  teamwork  (Human-Electronic 
Crew:  Can  we  trust  the  team?).  Cambridge,  UK,  27-30  September  1994. 


Overview: 

The  CAMA  system  is  designed  to  assist  the  crew  in  enhancing  of  the  situation  awareness  in  all 
mission  phases,  from  flight  planning  to  the  debriefing  after  landing.  CAMA  provides  several  types 
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of  operational  support  (e.g.,  preparation  and  execution  of  tactical  navigation).  Basic  services  are 
also  available  (e.g.,  check  list  routine  work,  fuel/load  management,  etc.) 

The  CAMA  program  consists  of  16  separate  modules  for  data  acquisition  and  data  control.  These 
modules  are  grouped  according  to  tracking  goals/tasks  (plan  generator  and  recognition  monitor); 
operator  state  (physio-behavioural  monitor),  world  state  (terrain,  aircraft  state  and  parameters  and 
environment  monitors),  adaptation  (pilot  intent  and  error  recognition,  terrain  interpreter),  and 
interaction  modules  (interface). 


Conclusions  for  IASs: 

1 .  It  is  recommended  that  functional  overlapping  should  be  avoided  between  modules  and  pre¬ 
existent  aircraft  monitoring  systems. 


3.3.2  Co-Pilote  Electronique  (France) 


Reference: 


Joubert,  T.,  Salle,  S.E.,  Champigneux,  G.,  Grau,  J.Y.,  Sassus,  P.,  and  Le  Doeuff,  H.  (1995).  The 
Copilote  Electronique  project:  First  lessons  as  explanatory  development  starts.  Proceedings  of  the 
3rd  international  workshop  on  human-computer  teamwork  (Human-Electronic  Crew:  Can  we  trust 
the  team?).  Cambridge,  UK,  27-30  September  1994. 
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Overview: 

The  Co-Pilote  Electronique  program,  which  was  initialized  in 
1986,  focused  on  artificial  intelligence  support  for  problem 
recognition  and  situation  assessment.  This  paper  describes 
the  first  lessons  learnt  as  a  new  phase  of  the  French  project 
"Copilot  Electronique"  (CE)  that  began  in  1994.  It  details  the 
development  of  the  system  and  compares  the  advantages  and 
drawbacks  of  existing  methodologies. 

According  to  Taylor  (1998),  the  French  Co-Pilote  Electronique 
programme  aimed  to  provide  cockpit  assistance  to  the  military  fast-jet  combat  pilot.  The  CE 
approach  was  to  provide  assistance  with  situation  assessment  and  planning  (with  multi-agent 
technology),  but  not  plan  execution  (according  to  roles/stages  model,  this  system  provided 
automation  of  information  acquisition,  analysis  and  presentation  and  solution  generation  but  no 
action).  Intent  recognition  and  intent  planning  are  performed  to  minimize  operator  workload.  The 
type  of  recommendations  given  by  the  system  was  modulated  by  the  current  situation  and  pilot 
mental  load.  The  pilot  interacted  with  the  system  through  a  “supervisor  expert  function”. 

Three  main  risks  were  identified  at  the  end  of  the  phase: 

1 .  It  was  not  clear  if  it  was  possible  to  capture  enough  expertise  to  create  real  assistance  for  pilot 
reasoning. 

2.  There  was  doubt  as  to  whether  knowledge-based  system  technology  was  mature  enough  for 
the  development  of  a  real  system. 

3.  It  was  still  not  clear  if  the  French  Aerospace  community  would  be  able  to  integrate  such  a  new 

concept  in  current  avionics  systems  design. _ 
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Conclusions  for  IASs: 

1 .  The  authors  found  that  by  the  end  of  the  first  phase,  conventional  knowledge  engineering 
techniques  using  questionnaires  and  interviews  were  not  sufficient  to  provide  implementable 
and  secured  knowledge  for  Pilot  aids. 

2.  Common  plans  and  goals  exchange  language  between  all  specific  assistance  modules  must 
be  carefully  defined  (a  technical  issue). 

3.  The  intent  planning  paradigm  proved  to  be  a  useful  unifying  technical  principle  to  facilitate 
architecture  design. 


3.3.3  Delegated  Systems:  Playbook  (United  States) 


Reference: 


Miller,  C.,  Funk,  H.,  Wu,  P.,  Goldman,  R.,  Meisner,  J.,  Chapman,  M.  (2005).  The  Playbook 
Approach  to  Adaptive  Automation,  In  Proceedings  of  the  Human  Factors  and  Ergonomics  Society's 
49th  Annual  Meeting,  Orlando,  FL 


Overview: 

Playbook  is  a  human-system  communication  tool  that  allows 
delegated  control  of  automation.  The  tool  is  based  on  a 
shared  model  of  the  tasks  in  a  domain.  This  shared  task 
model  provides  a  means  of  human-automation  communication 
about  plans,  goals,  methods  and  resource  usage,  a  process 
similar  to  referencing  plays  in  a  sports  team’s  playbook.  The 
Playbook  enables  operators  to  interact  with  subordinate 
systems  with  human  subordinates,  thus  allowing  for  adaptive 
automation.  This  approach  and  its  application  is  described 
through  an  ongoing  project  called  Playbook-enhanced  Variable  Autonomy  Control  System™  (P- 
VACS). 

Playbook  is  a  specific  method  of  implementing  a  delegation  interaction  which  can  be  divided  into 
two  components:  (1)a  hierarchical  task  model  that  is  compatible  with  levels  of  automation  (cf. 
Sheridan,  1987);  and  (2)  a  planning  mechanism  for  evaluating  existing  resources,  plan  validity,  and 
instantiating  the  task  models. 

A  shared  task  model  is  comprised  of  a  set  of  play  templates  are  generated  by  identifying  a  set  of 
common  tasks,  grouping  those  tasks  into  plays,  and  enabling  elements  such  as  time  and  location 
to  become  task  parameters. 

How  Playbook  works.  When  a  previously  defined  play  is  executed,  the  operator  can  select  a  play 
template  and  apply  the  parameter  values  as  appropriate  to  his/her  needs.  Both  the  operator  and 
the  automation  have  a  similar  model  of  the  sequence  of  tasks  to  execute  (the  shared  task  model). 

The  overall  Playbook  architecture  consists  of  three  components:  a  library  of  task  models;  a 
constraint-based  planning  engine;  and  an  OMI. 
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Conclusions  for  IASs: 

1 .  Findings  provide  support  for  allowing  the  tasking  of  multiple  agents  while  keeping  the 
supervisor  in  the  decision-making  loop,  without  increasing  supervisor  mental  workload.  It  also 
suggests  that  the  human  supervisor  can  adapt  successfully  to  unpredictable  changes  in  the 
environment. 

2.  Playbook  provides  a  complete  architecture  for  the  integration  of  human  input,  intelligent  a  priori 
planning,  reactive  planning  and  event  handling,  and  ongoing  vehicle  control  loops. 

3.  The  authors  recognize  that  new  methodologies  are  still  needed  to  build  more  extensive  task 
models.  For  instance,  Playbook  task  knowledge  should  arise  from  results  of  Cognitive  Work 
Analysis  of  a  task  domain  and  then  the  Playbook  architecture  (including  III  and  planning 
components)  can  be  used  to  produce  useful  task  timeline  inputs  for  a  constructive  simulation. 


Reference: 


Miller,  C.  and  Goldman,  R.  (1999).  Tasking  interfaces:  Associates  that  know  who's  the  boss.  In  J. 
Reising,  R.  M.  Taylor  and  R.  Onken  (Eds.).  The  human  electronic  crew:  The  right  stuff? 
Proceedings  of  the  4th  joint  GAF/RAF/USAF  workshop  on  human-computer  teamwork,  Kreuth, 
Germany  (Technical  report  AFRL-HE-WP-TR-1 999-0235  pp. 97-102).  Wright  Patterson  AFB,  OH: 
Air  Force  Research  Laboratory. 


Overview: 

This  paper  describes  the  techniques,  adapted  from  the  “associate”  (PA)  research,  used  for  the 
construction  of  tasking  interfaces.  They  present  initial  work  on  a  solution,  which  allows  human 
operators  to  interact  with  advanced  automation  at  various  levels.  According  to  this  model,  tasked 
systems  should  always  be  sub-ordinate,  but  must  know  enough  about  the  tasks  in  the  domain.  The 
authors  claim  that  instructing  these  “tasking  interfaces”  is  vastly  easier  than  instructing  traditional 
automated  systems.  Concepts  are  described  and  discussed  in  the  context  of  a  tasking  interface  for 
UAVs. 

Playbook  OMI 

This  is  an  interface  that  allows  the  operator  to  inspect  and  interact  with  the  system  (through  a  task 
model)  by  “calling  plays”  and  activating  tasks  at  various  levels  and  sub-levels.  Through  this 
interface,  the  operator  will  graphically  instruct  a  full  or  partial  plan  for  the  mission  by  specifying  the 
tasks  to  be  performed,  or  goals  to  be  accomplished  by  the  system  (Figure  6). 

Playbook  Framework: 

The  framework  is  composed  of  four  primary  components: 

1.  Playbook  OMI 

2.  Mission  Analysis.  A  projective  planning  system  which  is  capable  of  understanding 
instructions  from  the  operator  through  the  OMI. 

3.  Event  Handling.  All  accepted  plans  are  passed  from  the  mission  analysis  module  to  “even 
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handling”  where  plans  can  be  adjusted  in  real-time. 

4.  Control  algorithms.  Executes  the  instructions. 

This  framework  is  based  upon  and  interacts  with  a  Shared  Task  Model  Infrastructure,  which  can 
facilitate  human-system  coordination. 


Conclusions  for  IASs: 

1 .  The  authors  stress  that  a  usability  evaluation  of  the  tasking  interface  GUI  (and  all  system 
interfaces)  is  required. 

2.  The  authors  warn  that  tasking  interfaces  should  not  rely  on  a  predefined  set  of  task  models, 
but  dynamic  ones.  The  operator  should  be  able  to  create  novel  tasks  and  to  store  components 
of  models  which  are  useful. 

3.  The  authors  acknowledge  that  their  task  network  representation  is  weak  in  its  coding  of  goals, 
which  are  seen  as  a  critical  component  of  any  tasking  interface. 

4.  Operators  need  sufficient  training  for  interacting  with  the  tasking  interface. 

5.  A  delegated  interface  may  increase  operator  acceptance;  that  is,  by  enabling  a  system  to 
behave  more  like  an  intelligent  subordinate,  operators  may  be  more  tolerant  of  their 
weaknesses  and  more  acceptable  of  their  capabilities  in  a  controlled  setting. 


Reference: 


Miller,  C.  (2005).  Using  Delegation  as  an  Architecture  for  Adaptive  Automation.  Technical  Report 
(No.  AFRL-H E-WP-TP-2005-0029). 


Overview: 

A  3D  model  framework  for  adaptive  automation,  referred  to  as  "Levels  of  Delegation",  is  described. 
Delegation  implies  that  a  subordinate  is  given  the  responsibility  to  perform  a  task  (with  its 
subtasks),  along  with  some  authority  to  decide  how  to  perform  that  task,  as  well  as  access  to 
resources  with  some  authority  to  decide  how  to  use  them  to  perform  the  task.  This  paper  describes 
the  use  of  this  framework  within  an  application  called  Playbook. 

The  “Delegation  Framework”  has  three  dimensions,  AAA:  Level  of  Authority,  Level  of  Abstraction 
and  Level  of  Aggregation.  These  dimensions  define  a  Delegation  Space  of  human-automation 
relationships  within  which  delegation  occurs  and  can  be  characterized.  The  three  scales  must  be 
used  to  specify  four  variables  which  define  the  delegation  space:  the  level  of  abstraction  and  the 
level  of  authority  on  it,  and  the  level  of  aggregation  and  the  level  of  authority  on  it. 

Below  is  a  description  of  each  dimension: 

•  Levels  of  Authority.  Full,  inform,  override,  approval,  recommend,  monitor,  none. 

•  Level  of  Abstraction.  Automation  can  have  responsibility  for  higher-  or  lower-level  tasks  within 
the  task  hierarchy. 

•  Level  of  Aggregation.  Identifies  how  much  (and/or  which  type)  of  resource  each  actor  is 
authorized  to  use. 
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Conclusions  for  IASs: 

1 .  All  three  dimensions  may  not  be  available  or  relevant  to  every  system  or  every  interaction,  but 
the  authors  advocate  that  the  model  needs  to  be  rich  enough  to  encompass  them. 
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Figure  5:  Playbook  tasking  interface  for  a  military  fast-jet  ground  attack  mission. 


3.3.4  Intelligent  Classroom 


Reference: 


Franklin,  D.,  Budzik,  J.,  and  Hammond,  K.  (2002).  Plan-based  Interfaces:  Keeping  Track  of  User 
Tasks  and  Acting  to  Cooperate.  In  IUI’02,  January  13-16,  2002. 


Overview: 

This  paper  describes  the  concept  of  an  Intelligent  Classroom,  which  consists  of  a  computer  system 
that  dynamically  adapts  to  operator  actions  and  inputs  (gesture  and  voice)  in  a  classroom 
environment  (i.e.,  controls  camera,  automatic  presentation  slide-switcher).  The  algorithms  are 
goal-based  and  driven  by  task  recognition. 

Intelligent  Classroom:  The  1C  is  an  automated  lecture  facility  prototype  that  serves  as  its  own 
audio/visual  assistant.  The  operator  (e.g.,  speaker),  provides  a  presentation,  and  the  Classroom 
watches  and  listens,  and  when  appropriate,  assists  will  provide  assistance.  The  1C  keeps  track  of 
various  activities  pursued  by  the  speaker  as  well  as  its  own  activities  in  control  of  its  various _ 
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autonomous  components. 

The  representation  is  used  three  ways  to  accomplish  a  goal: 
plan  execution  (execute  a  plan  to  achieve  a  goal),  plan 
recognition  (match  the  operator’s  actions  to  a  set  of  known 
plans),  and  projection  (follows  the  operator’s  plan  and  project 
future  actions).  A  set  of  agents  are  used  to  monitor,  recognize, 
and  execute  some  plan  to  accomplish  an  operator  goal. 

The  system  is  based  on  the  principle  that  the  world  is 
composed  of  a  series  of  processes.  A  process  is  a  single 
agent  that  executes  a  sequence  of  actions.  It  is  composed  of  one  or  more  discrete  steps,  each  of 
which  specifies  a  number  of  continuous  actions  and  a  number  of  discrete  events.  The  processes 
are  designed  such  that  the  Classroom  can  essentially  use  the  same  algorithm  for  executing  a 
process  that  it  used  for  observing  the  operator  as  the  operator  executes  a  process. 

To  alter  the  algorithm  so  that  the  Classroom  can  observe  the  operator  and  to  follow  along  with  the 
operator’s  plans,  only  a  portion  of  the  first  step  needs  to  be  changed.  Rather  than  performing  the 
primitive  actions  that  are  a  part  of  the  step,  the  Classroom  performs  “observation”  actions  that 
complement  the  primitive  actions. 

The  Process  manager  continually  steps  through  its  set  of  processes  to  keep  them  synchronized 
with  the  operator  and  revises  the  set  of  processes  when  required. 

Human-machine  cooperation.  The  operator,  in  executing  part  of  a  plan,  expects  the  Classroom  to 
do  its  part  of  the  plan.  A  plan  is  a  set  of  processes  (often  to  be  executed  by  a  number  of  different 
agents)  that  when  executed  together  successfully,  accomplish  some  goal.  In  the  Classroom,  most 
plans  have  one  process  executed  by  the  operator  and  one  or  two  processes  executed  by  the 
Classroom.  This  definition  makes  explicit  the  presence  of  other  agents  or  exogenous  events.  In  the 
Classroom,  these  plans  attempt  to  express  a  common  understanding  of  how  a  speaker  and  an 
audio/visual  assistant  interact. 


Conclusions: 

1 .  The  more  the  system  understands  its  operators  and  their  tasks,  the  more  useful  the  system  will 
be. 

2.  The  same  techniques  implemented  in  the  Intelligent  Classroom  can  be  applied  to  a  broad 
range  of  interactive  applications.  Refer  to  the  paper  for  details  on  how  to  implement 
techniques. 

3.  The  system  should  understand  the  operator’s  actions  in  the  context  of  what  it  believes  the 
operator  is  doing. 

4.  The  ability  to  provide  reason  to  the  operator’s  activity  is  crucial  to  the  implementation  of  an 
intelligent  operator  interface. 

5.  Plan  generation  and  recognition  are  a  promising  means  of  adaptive  automation  and  estimating 
pilot  intent. 

6.  Human-machine  cooperation  can  be  achieved  by  allowing  an  operator,  when  executing  a  part 
of  a  plan,  to  expect  a  system  to  help  in  executing  that  part  of  the  plan.  A  plan  is  a  set  of 
processes  (often  to  be  executed  by  a  number  of  different  agents)  that  when  executed  together 
successfully,  accomplish  some  goal.  Plans  attempt  to  express  a  common  understanding  of 
how  a  speaker  and  an  audio/visual  assistant  interact. 
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3.3.5  Lookout 


Reference: 


Horvitz,  E.  (1999).  Principles  of  mixed-initiative  user  interfaces.  In  Proceedings  of  ACM 
Conference  on  Human  Factors  in  Computing  Systems  (CHI'99),  pp.  155-166. 


Overview: 

The  authors  review  principles  for  directly  manipulating 
automation  and  machine  learning.  These  principles  are 
highlighted  in  terms  of  the  program  called  LookOut,  an 
automated  system  for  scheduling  and  meeting  management. 

LookOut:  Is  a  program  that  automatically  populates  meeting 
request  information  based  on  an  email  message  text  in  the 
body  and  subject. 

Initiation  of  Lookout:  The  system  can  be  initiated  either  by  the 
human  (clicking  on  icon  or  when  prompted  by  the  system)  or  automatically  by  the  system  based  on 
a  goal-based  user  model. 

Direct  manipulation.  The  operator  communicates  directly  with  the  system  through  an  animated 
widget. 

User  model.  The  user  model  is  based  on  a  “function  of  an  inferred  probability”  that  the  operator 
has  a  goal  of  performing  scheduling  and  calendaring  operations. 

Confidence  estimation.  The  level  of  automation  (initiation  and  action)  is  based  on  the  system’s 
uncertainty  of  the  operator’s  goals  which  is  based  on  the  user  model.  The  authors  applied 
probabilistic  models  of  an  operator’s  goals.  This  is  used  to  perform  real-time  inferences  about  the 
probability  of  alternate  feasible  goals  by  monitoring  the  current  program  context,  and  the  operator’s 
sequence  of  actions  and  choice  of  words  used  in  a  query.  Bayesian  network  models  were  partially 
used  to  for  a  base  for  the  confidence  estimation  algorithms. 

Displaying  automation  uncertainty.  The  level  of  uncertainty  about  the  operator’s  goals  is  displayed 
to  the  operator  via  visual  indicators.  At  high  levels  of  certainty,  a  character  appears  and  indicates 
that  it  has  readied  a  calendar  view  to  show  the  operator  or  has  created  a  tentative  appointment 
before  displaying  the  results.  At  lower  levels  of  confidence,  LookOut  inquires  about  the  operator’s 
interest  in  either  seeing  the  calendar  or  scheduling  an  appointment,  depending  on  the  system’s 
analysis  of  the  message  being  viewed. 

Automated  tasks.  The  decision  of  initiating  automation  is  based  on  whether  an  agent  believes  it  will 
have  greater  expected  value  than  inaction  for  the  operator,  taking  into  consideration  the  costs, 
benefits  and  uncertainties  in  the  operator’s  goals.  Refer  to  the  paper  for  implementation  details. 

Timing  of  prompting  the  initiation  of  automation.  Automation  and  alerts  of  initiating  automation  is 
based  on  models  of  attention  that  consider  the  temporal  pattern  of  an  operator’s  focus  of  attention 
(timing  model). 

Machine  learning.  The  system  is  designed  to  continue  to  learn  from  operators  through  caching 
operator  behaviour  with  the  system  and  by  the  operator  specifying  a  policy  for  continual  learning 
(e.g.,  set  system  to  cache  behaviour  at  particular  times). 

The  authors  recommend  considering  several  critical  factors  when  implementing  integration  of 
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automated  services  with  direct  manipulation  interfaces,  as  discussed  below. 


Conclusions  for  IASs: 

1 .  Uncertainty  about  an  operator’s  goals  can  provide  good  input  for  inferring  about  an  operator’s 
intentions  to  perform  an  operation.  Computers  are  often  uncertain  about  the  goals  and  the 
current  focus  of  attention  of  an  operator.  In  many  cases,  systems  can  benefit  by  employing 
machinery  for  inferring  the  uncertainty  about  an  operator’s  intentions  and  focus. 

2.  Considering  the  status  of  an  operator’s  attention  in  the  timing  of  services.  Systems  (or  agents) 
could  use  models  of  attention  and  consider  the  costs  and  benefits  of  deferring  action  to  a  time 
when  the  automation  will  be  less  distracting  to  the  operator. 

3.  Context-dependent  automation.  Automated  functions  should  be  applied  in  a  context-relevant 
manner  based  on  uncertainty  in  an  operator’s  goals  and  attention. 

4.  The  system  should  resolve  uncertainties  through  a  dialog  with  the  operator.  If  a  system  is 
uncertain  about  an  operator’s  intentions,  it  should  be  able  to  engage  in  a  dialog  with  the 
operator,  considering  the  costs  of  potentially  bothering  an  operator  needlessly. 

5.  Direct  invocation  and  termination  of  automation  should  be  provided.  Efficient  means  should  be 
provided  which  operators  can  directly  invoke  or  terminate  the  automated  services. 

6.  Operators  should  have  an  efficient  means  to  modify  automation  behavior.  Agents  should  be 
designed  so  that  operators  can  complete  or  refine  an  analysis  provided  by  an  agent. 

7.  Agent-operator  interaction  should  employ  socially  appropriate  behaviors.  An  agent  should  be 
designed  to  behave  in  a  way  that  matches  social  expectations. 

8.  Recent  operator  interactions  with  the  system  should  be  saved.  Systems  should  maintain  a 
memory  of  recent  interactions  with  operators  and  provide  mechanisms  that  allow  operators  to 
make  references  to  objects  and  services  included  in  “shared”  short-term  experiences. 

9.  Learning  by  observing  operator  behavior.  Systems  should  be  designed  so  that  they  continue  to 
learn  about  an  operator’s  goals  and  needs. 


3.4  Intelligent  Adaptive  Hybrid  Frameworks 

This  section  reviews  the  following  I  AH  frameworks: 

•  Cognitive  Cockpit; 

•  Associate  Systems  Technology  (Pilot’s  Associate  and  Rotorcraft  Pilot’s  Associate); 

•  Cockpit  Assistant  System  (CASSY);  and, 

•  DRDC  Toronto’s  Intelligent  Adaptive  Interface  project  for  Uninhabited  Aerial 
Vehicles. 

3.4.1  Cognitive  Cockpit  (United  Kingdom) 

The  Cognitive  Cockpit  was  originally  a  large  multi-disciplinary  project  funded  by  the 
Ministry  of  Defence  (United  Kingdom),  and  conducted  at  the  Defence  Evaluation  and 
Research  Agency  (DERA),  and  then  at  QinetiQ,  that  is  concerned  with  quantifying  the 
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effectiveness  of  pilot  aiding  designed  to  increase  mission  effectiveness  and  safety.  The 
objective  of  the  work  was  to  specify  the  cognitive  requirements  for  building  the  next 
generation  of  cockpit  intelligent  aiding  systems  for  use  in  2010-2015  time  scales.  Since  2000, 
the  programme  has  continued  under  the  funding  of  DARPA  (United  States),  but  has  changed 
its  emphasis  to  the  control  of  multiple  Uninhabited  Combat  Air  Vehicles  (UCAVs). 

The  goals  of  the  Cognitive  Cockpit  will  be  achieved  by  developing  an  integrated  system 
within  a  human-centred  approach  that  keeps  the  operator  in  charge.  This  approach  focuses  on 
the  pilot’s  requirement  to  be  in  control  of  the  system  and  not  be  overwhelmed  with  system 
control  information.  Thus,  the  fundamental  assumption  of  this  research  programme  is  to  allow 
pilots,  either  airborne  or  on  the  ground  controlling  a  UCAV,  to  concentrate  their  skills 
towards  the  relevant  critical  mission  event,  at  the  appropriate  time,  and  to  the  appropriate 
level. 

To  provide  a  principled  development  of  intelligent  aiding  with  the  required  levels  of  pilot 
control,  the  project  team  established  a  guiding  framework  for  “Cognitive  Control”  in  the 
Cognitive  Cockpit  programme  (Taylor,  1997;  Taylor  and  Finnie,  1997;  Taylor  and  Reising, 
1999).  This  framework  is  based  on  the  concepts  and  implications  of  Perceptual  Control 
Theory  (Powers,  1973;  Taylor,  1997),  and  on  the  extant  theory  of  Cognitive  Control  of 
Complex  Systems  (Rasmussen,  1986,  1993;  Brehmer,  1992;  and  Hollnagel,  1997).  By 
highlighting  the  importance  of  cognitive  control,  a  more  direct  and  systematic  consideration 
of  the  cognitive  engineering  and  control  issues  can  be  achieved.  For  example: 

•  The  incorporation  of  the  ability  to  track  the  operator’s  goals  and  plans  (e.g.  the 
difference  between  current  and  desired  states)  and  to  infer  the  intent  of  the  operator; 

•  The  use  of  abstraction  hierarchies  and  system  aggregation  methods  during  task 
decomposition  in  order  to  determine  important  interactions  and  emergent  properties 
within  the  knowledge  base; 

•  The  importance  of  information  utility  in  the  design  process  (e.g.,  a  focus  on  the 
information  used,  rather  than  the  resultant  action); 

•  The  importance  of  error  diagnosis  and  rectification; 

•  The  enhancement  of  system  stability  through  the  balance  of  feed-back  (i.e.,  reactive) 
and  feed-forward  (i.e.,  proactive)  control  information; 

•  The  recognition  of  differences  in  cognitive  control  strategies  between  skill,  rule  and 
knowledge-based  levels  of  performance  (Rasmussen,  1986); 

•  The  incorporation  of  planning  horizons  (e.g.,  scrambled,  opportunistic,  tactical  and 
strategic;  Hollnagel,  1997)  into  cognitive  control  strategies;  and, 

•  The  use  of  intelligent  aiding  to  critique  operator  performance  and  to  prevent  cognitive 
bias  and  other  forms  of  human  error. 

The  Cognitive  Cockpit  program  brought  together  DERA’s  experience  with  systems  for  state 
monitoring  pilot,  capability  in  building  Knowledge-Based  Systems  for  decision  support,  and 
expertise  in  cognitive  engineering  and  cockpit  human  factors  integration.  The  Cognitive 
Cockpit  comprises  four  functional  modules: 

•  Cognition  Monitor  (CogMon).  This  module  is  concerned  with  on-line  analysis  of  the 
psychological,  physiological  and  behavioural  state  of  the  pilot.  Primary  functions  of 
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this  system  include  continuous  monitoring  of  workload,  and  inferences  about  current 
attentional  focus,  ongoing  cognition  and  intentions.  It  also  seeks  to  detect  dangerously 
high  and  low  levels  of  arousal.  Overall,  this  system  provides  information  about  the 
objective  and  subjective  state  of  the  pilot  within  a  mission  context.  This  information 
is  used  in  order  to  optimise  pilot  performance  and  safety,  and  provides  a  basis  for  the 
implementation  of  pilot  aiding; 

•  Situation  Assessment  Support  System.  This  module  is  concerned  with  on-line  mission 
analysis,  aiding  and  support  provided  by  intelligent  software.  This  system  is  privy  to 
the  current  mission,  aircraft  (e.g.  heading,  altitude  and  threat)  and  environmental 
status,  and  is  also  invested  with  extensive  a  priori  tactical,  operational  and  situational 
knowledge.  Overall,  this  system  provides  information  about  the  objective  state  of  the 
aircraft  within  a  mission  context,  and  uses  extensive  knowledge-based  systems  in 
order  to  aid  and  support  the  pilot; 

•  Tasking  Interface  Manager.  This  module  is  concerned  with  on-line  analysis  of 
higher-order  outputs  from  CogMon,  SASS,  as  well  as  other  aircraft  systems.  A  central 
function  for  this  system  is  maximisation  of  the  goodness  of  fit  between  aircraft  status, 
‘pilot- state’  and  tactical  assessments  provided  by  the  SASS.  These  integrative 
functions  mean  that  this  system  must  be  able  to  influence  the  prioritisation  of  tasks 
and,  at  a  logical  level,  determine  the  means  by  which  information  is  communicated  to 
and  from  the  pilot.  Overall,  this  system  allows  pilots  to  manage  their  interaction  with 
the  cockpit  automation,  by  way  of  control  over  the  allocation  of  tasks  to  the 
automated  systems;  and, 

•  Cognitive  Cockpit  (Cogpit).  This  module  is  concerned  with  the  specification  and 
provision  of  a  proof-of-concept  simulation  environment,  including  the  form  and 
function  of  a  future  cockpit  in  which  Situation  Assessment  Support  System, 

Cognition  Monitor  and  Tasking  Interface  Manager  modules  will  be  implemented, 
tested  and  validated.  In  doing  so,  there  is  a  requirement  to  utilise  existing  Human 
Factors  analysis  methods  and  Human-Computer  Interaction  guidelines. 
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Reference: 


Taylor,  R.M.  (2001 ).  Technologies  for  supporting  human  cognitive  control.  In  Proceedings  of  the 
RTO  HFM  Specialists’  Meeting  on  Human  Factors  in  the  21st  Century,  Paris,  France,  11-13  June 
2001. 
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Overview: 

A  proof-of-concept  demonstration  of  Cognitive  Cockpit 
technologies  was  undertaken  by  DERA  that  sought  to  couple 
on-monitoring  of  pilot  functional  state  assessment, 
environment  and  mission  plan.  The  aim  was  to  allow  the  pilot 
in  control  of  the  aircraft,  or  the  operator  in  control  of  an 
uninhabited  air  vehicle  to  “concentrate  on  their  skills  towards 
the  relevant  critical  mission  event,  at  the  appropriate  time,  to 
the  appropriate  level”.  The  framework  that  formed  the  basis  of 
the  Knowledge-Based  system  (roles),  and  adaptation  (automation)  was  a  feed-forward  (operator 
and  system),  and  feed  backward  (system)  control  task  architecture.  CommonKADS  knowledge 
engineering  methodology  led  to  the  development  of  six  knowledge-level  models:  organisational, 
task,  agent,  knowledge,  communication  and  design  models.  These  models  were  used  as  a  basis 
for  the  development  of  a  multi-agent  COGPIT  system  involving  the  following  COGPIT  agent 
architecture: 

Cognition  Monitor.  Monitors  pilot  functional  state  (level  of  arousal  and  workload)  which  was  based 
on  the  cognitive  model. 

Situation  Assessor.  Monitors  environmental  and  aircraft  state  and  recommends  actions.  It  is  based 
on  the  organization,  task  and  knowledge  models.  Provides  information  about  the  state  of  the 
aircraft  within  the  context  of  a  mission  and  supports  the  decision-making  process  (considered  a 
knowledge-based  decision-support  system). 

Tasking  Interface  Manage.  Implements  adaptation  (assignment  of  automation  to  the  human  or 
system  agent)  and  is  based  on  COGMON  and  SASS  (i.e.,  maximum  goodness  of  fit  between 
aircraft  status,  pilot  state  and  tactical  assessments). 

Pilot  Authorizing  and  Control  Tasks  (PACT).  PACT  is  operator  initiated  decision  aiding  and 
implementing  automation  (pilot  control  of  tasks).  Based  on  the  roles/stages  of  information 
processing. 

Dynamic  adaptive  interface.  Automatically  assigns  roles  to  the  system  or  operator  according  to 
operator  agreed,  context-sensitive  adaptive  rules. 

The  aim  of  the  COGPIT  architecture  was  to  increase  system  adaptiveness  by  enabling  changes  to 
be  made  to  the  mission  plan  in  response  to  changes  in  the  situation.  The  COGPIT  monitored  three 
aspects  of  the  situation:  the  pilot  to  take  account  of  his  physiological  and  cognitive  state,  the 
environment,  both  external  to  the  aircraft  and  the  aircraft  systems,  and  the  mission  plan  to  indicate 
current  and  future  pilot  actions.  Information  from  monitoring  the  environment,  the  mission  plan  and 
the  pilot  provided  inputs  into  the  processes  of  re-planning  the  mission,  automating  tasks,  deciding 
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automation  and  configuring  the  cockpit.  These  processes  then  provided  input  into  updating  the 
mission  plan. 


Conclusions  for  IASs: 

1 .  lAHs  should  be  based  on  an  Operator-Centred  Design  (OCD)  process. 

2.  CommonKADS  models  proved  to  be  a  useful  knowledge  acquisition  technique  for  the 
development  of  the  SASS  model.  The  PC  PAC  and  MetaPAC  toolsets  were  essential  in 
supporting  the  acquisition  and  modeling  processes. 

3.  Knowledge  acquisition  techniques,  based  on  a  Goals,  Means  Task  Analysis  methodology, 
were  necessary  for  developing  the  task-based  TIM,  and  particularly  useful  for  on-line  KBS 
support  for  pilot  re-planning  tasks, 

4.  Timing  is  critical  for  contextual  KBS  advice  to  be  effective. 

5.  Intelligent  aiding  can  be  successfully  implemented  if  it  has  a  well-developed  multi-system 
architecture  based  on  an  operator-centred  design  methodology. 


Taylor,  R.  M.,  Bonner.  M.  C.,  Dickson,  B.,  Howells,  H.,  Miller,  C.  A.,  Milton,  N.,  Pleydell-Pearce,  K., 
Shadbolt,  N.,  Tennison,  J.,  &  Whitecross  (2003).  Cognitive  Cockpit  Engineering:  Coupling 
Functional  State  Assessment,  Task  Knowledge  Management,  and  Decision  Support  for  Context- 
Sensitive  Aiding.  In  M.  D.  McNeese  &  M.  Vidulich  (Eds.).  Cognitive  systems  engineering  in  military 
aviation  environments:  Avoiding  cogminutia  fragmentosa!  (pp.  253-312).  Human  Systems 
Information  Analysis  Center  State-of-the-art  Report  02-01.  Wright-Patterson  Air  Force  Base,  OH: 
Human  Systems  Information  Analysis  Center. 


Overview: 

Describes  the  Cognitive  Cockpit  research  program  that  seeks  to  couple  pilot  functional  state 
assessment,  knowledge-based  systems  for  situation  assessment,  and  decision  support,  with 
concepts  and  technologies  for  adaptive  automation  and  cockpit  adaptive  interfaces.  The  authors 
advocate  for  a  functional  integration  approach  to  system  development  where  several  functional 
components  can  collectively  perform  many  of  the  same  behaviours  as  the  pilot,  and  of  cognitive 
control  between  the  pilot  and  the  intelligent  aiding  systems. 

The  authors  also  present  a  summary  of  the  methods,  tools,  and  techniques  used  on  the  COGPIT 
project  in  the  phases  of  the  development  of  the  COGPIT  systems,  including  cognitive  engineering 
(see  Table  8.6  on  p.  305).  Work  to  date  has  provided  mission-based  functional  decomposition, 
cognitive  task  analysis,  knowledge  acquisition  and  modeling,  interface  prototyping,  initial  proof-of- 
concept  simulation,  and  cognitive  story-board  evaluation. 

To  determine  required  levels  of  pilot  control,  the  authors  based  their  framework  on  concepts  and 
implications  of  Perceptual  Control  Theory  and  the  theory  of  cognitive  control  of  complex  systems; 
this  work  was  also  influenced  by  the  Information  Processing  (IP)  theory  load  under  time  pressure 
and  DERA’s  work  on  cognitive  streaming. 

COGMON.  In  order  to  provide  a  real-time  model  about  the  cognitive-affective  state  of  a  pilot,  four 
principle  sources  of  information  are  available:  physiology,  behaviour,  subjective  and  context  states. 
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•  Physiological  Measures:  Measures  include  heart  rate,  respiration  rate,  electromyogram, 
electrodermal  activity,  skin  temperature,  electro-EEG,  eye-movement  activity  and  blink  rate. 
Since  many  physiological  measures  are  correlated,  COGMON  uses  various  mathematical 
tools  aimed  at  uncoupling  correlations  between  its  incoming  physiological  variables. 

•  Behavioural  Measures:  Behavioural  data  provide  information  that  can  be  used  to  make 
inferences  about  cognitive  states.  COGMON  uses  a  pre-existing  database  and  probability 
theories  about  which  combination  of  procedures  attempt  to  infer  pilot  intent.  It  is  also 
contextually  sensitive. 

•  Subjective  Measures:  These  measures  are  used  in  to  get  explicit  feedback  from  the  pilot  (1 ) 
during  the  task  and  (2)  during  system  development. 

•  Contextual  Measures:  to  provide  COGMON  a  context  in  which  to  interpret  the  incoming 
information.  Some  of  these  measures  include  ambient  noise,  luminance,  aircraft  parameters, 
etc. 

Operator  customization.  COGMON  has  a  database  that  holds  information  about  each  operator 
to  provide  operator  specific  assistance. 

COGMON  architecture  includes  individual  components  that  can  also  function  in  isolation  which 
can  be  adapted  for  other  systems. 

SASS,  the  Situation  Assessor,  handles  situation  assessment  on  a  task-by-task  basis  with  no 
separate  module  or  agent  to  perform  the  assessment  of  the  situation.  This  approach  allows  the 
system  to  assess  the  situation  within  the  context  of  the  task  being  performed  and  not  as  a 
separate  activity  (similar  to  what  expert  human  operators  do).  CommonKADS  was  used  to 
develop  this  module. 

TIM,  the  overall  architecture  of  an  adaptive  cockpit  involves  12  functions,  with  a  flow  of 
information  and  control  across  the  functions  (see  p289  for  chart).  This  module  is  based  on  a 
“Shared  Task  Model”  to  code,  track  and  dynamically  modify  operator’s  goals  and  plans.  This  is  a 
“delegation  module”  based  on  Playbook  principles.  The  task  model  used  for  COGPIT  uses  three 
task  categories:  generic,  mission  specific,  and  specific  tasks. 


Conclusions  for  IASs: 

1 .  The  authors  advocate  for  a  functional  integration  approach  to  system  development  where 
several  functional  components  can  collectively  perform  many  of  the  same  behaviours  as  the 
pilot  and  of  cognitive  control  between  the  pilot  and  the  intelligent  aiding  systems. 

2.  The  COGPIT  project  demonstrated  that  functional  analysis  of  cognitive  work  can  be  used  to 
provide  the  foundations  for  the  development  and  implementation  of  cognitive  technologies. 

3.  It  was  also  demonstrated  that  cognitive  work  analysis  seems  particularly  promising  in  providing 
a  broad  set  of  models  and  tools  for  human  systems  analysis,  based  on  a  high-level  functional 
analysis. 

4.  On-line  operator  functional  state  assessment  appears  to  be  feasible  with  current  computing 
power  and  may  be  useful  to  provide  information  for  the  dynamic  allocation  of  automation. 

5.  Knowledge  engineering  methodology  can  be  used  to  provide  on-line  knowledge-based 
systems  to  support  operator  re-planning  tasks. 

6.  IAH  system  OMIs,  tasks,  and  automation  can  be  managed  by  a  tasking  interface  system 
based  on  a  shared  task  model. 
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Reference: 


Bonner,  M.C.,  Taylor,  R.  M.,  Fletcher,  K.,  and  Miller,  C.  (2000).  Adaptive  automation  and  decision 
aiding  in  the  military  fast-jet  domain.  In  proceedings  of  the  conference  on  Human  Performance, 
Situation  Awareness  and  Automation:  User  centred  design  for  the  new  Millennium. 


Overview: 

This  paper  presents  the  operation  and  technical  development  of  the  Tasking  Interface  Manager 
component  of  the  Cognitive  Cockpit.  The  TIM  utilised  input  from  the  Situation  Assessment  Support 
System  and  the  Cognition  Monitor  to  adaptively  present  information  and  adaptively  automate  tasks 
according  to  the  situational  context  and  the  pilot's  internal  state.  The  goal  of  TIM  is  to  reduce  task 
and  cognitive  load  on  aircrew.  The  main  feature  of  the  TIM  is  a  shared  mental  model,  the  ability  to 
track  goals,  plans  and  tasks,  and  the  ability  to  communicate  intent  about  the  mission  plan.  The 
objective  of  the  TIM  is  to  allow  aircrew  to  retain  executive  control  of  aircraft  and  mission 
parameters  in  conjunction  with  the  assistance  of  adaptive  automation. 

TIM  involves  the  following  characteristics: 

1 .  A  shared  task  model:  The  TIM  was  based  on  a  task  model  to  help  encode,  track  and  model  the 
operator’s  goals  and  plans  and  ensure  that  they  are  highly-coordinated  with  the  system.  Three 
task  categories  were  used  in  the  COGPIT  framework:  1 )  task  generic  (tasks  that  never 
change);  2)  mission  specific  (tasks  that  change  with  the  mission  but  are  constant  within  the 
mission);  and  3)  task  specific  (tasks  which  change  within  the  mission). 

2.  Task  Tracking  Capabilities:  A  need  for  a  Full  Goal  Plan  Tracking  (GPT)  capability  was 
identified  during  the  knowledge  acquisition  process.  It  was  realized  that  the  system  should  be 
able  to  track  any  task  undertaken  by  the  pilot.  Implementation  of  this  system,  however,  is 
currently  limited  by  funding  resources  (as  of  year  2000). 

3.  Communication  about  Intent:  The  goal  of  this  approach  is  to  allow  the  human  operator  to 
communicate  tasking  instructions  (i.e.,  delegated  automation)  in  the  form  of  desired  goals, 
tasks,  partial  plans  or  constraints.  These  tasking  instructions  should  be  developed  in 
accordance  with  task  structures  defined  in  the  shared  task  model. 

4.  Adaptive  Automation:  Adaptive  automation  is  controlled  by  the  Plot  Authorization  and  Control 
of  Tasks  system,  an  operator-initiated  allocation  of  automation  (pilot  control  of 
tasks/automation).  The  allocation  of  automation  is  defined  by  the  level  of  authority  the  pilot  has 
over  the  initiation  of  automation  and  the  level  of  system  autonomy. 


Conclusions  for  IASs: 

1 .  The  authors  declare  that  tasks  (or  functions)  need  to  be  continuously  tracked  according  to  the 
state  of  the  mission  plan  in  order  for  the  system  to  determine  the  information  and  automation 
needs  of  the  operator. 

2.  To  maintain  operator  situational  awareness,  tasks  should  be  tracked  explicitly  (e.g.,  by  asking 
the  operator  for  input  or  making  the  system  state  visible  to  the  operator),  especially  in  high- 
criticality  environments. 

3.  The  allocation  of  tasks  to  the  system  should  be  controlled  by  the  operator.  This  allocation  can 
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be  controlled  by  the  operator  in  several  ways:  1 )  pre-set  operator  preferred  defaults;  2) 
operator  selection  during  pre-mission  planning;  3)  allocation  by  the  operator  during  in-mission 
re-planning;  and  4)  automatically  allocated  according  to  operator  agreed,  context-sensitive 
adaptive  rules. 

4.  The  automation  of  tasks  and  their  allocation  can  be  provided  by  a  tasking  interface  system 
based  on  a  shared  task  model.  The  use  of  a  tasking  interface  can  allow  an  operator  executive 
control  of  the  system  and  mission  while  enabling  almost  full  autonomy  for  an  aiding  agent. 


Taylor,  R.  M.  (1998).  The  human-electronic  crew:  Human-computer  collaborative  team  working. 
Proceedings  of  the  1st  NATO  RTO  Human  Factors  and  Medical  Panel  Symposium  on 
Collaborative  Crew  Performance  in  Complex  Operational  Systems,  Edinburgh,  UK,  20-22  April 
1998. 


Overview: 

This  paper  describes  the  concept  of  Human-Electronic  Crew  (HEC)  teamwork,  whereby  the 
electronic  crew  member  (the  electronic  support  system)  acts  as  an  associate  or  an  assistant, 
sharing  responsibility,  authority  and  autonomy  over  many  cockpit  tasks.  Various  methods  are 
suggested  for  ensuring  that  the  relationship  between  the  operator  and  the  system  is  flexible  and 
adaptive  including:  in-flight  situation  assessment  and  re-planning  (of  goals),  cognitive  modelling, 
human  intent  inferencing  and  error  recognition  (tracking  of  tasks),  and  the  use  of  complex 
knowledge  engineering  and  reasoning  logic  processes. 

Intelligent  aiding  systems  can  be  distinguished  in  terms  of  three  main  types  (all  three  types  can 
and  should  work/co-operate  together): 

•  Assistants:  perform  specific  tasks  when  asked,  using  basic  task  and  situational 
knowledge  (e.g.,  automation  of  a  task  such  as  auto-pilot). 

•  Associates:  recognize  that  the  operator  needs  assistance,  using  complex  task  and 
situational  knowledge,  and  basic  operator  co-ordination  knowledge  (i.e.,  allocates 
automation  based  on  an  operator  and  situational  model) 

•  Coaches:  both  aids  and  instructs  to  assist  the  operator  better,  using  complex  task, 
situation,  operator  and  co-ordination  knowledge  (e.g.,  decision  aid;  dynamic  function 
allocation). 

This  paper  compared  the  Co-Pilote  Electronic,  CASSY/CAMMA  and  Pilot’s  Associate  project  in 
implementation  strategy. 

1 .  Differences  in  approach  lie  in  how  the  automation  is  initiated  (who  has  control)  between  CE 
and  PA.  The  CE  approach  emphasizes  pilot  involvement  and  judgment  (pilot  control)  whereas 
the  PA  objective  is  a  full  associate  relationship  (sharing  of  control). 

2.  CE  and  CAMMA  projects  are  more  technically  realistic  (it  is  difficult  to  implement  a  full 
associate  relationship  because  the  system  is  expected  to  perform  at  the  same  level  as  the 
human  where  humans  can  make  leaps  of  abstraction  and  intuition,  producing  new  solutions  to 
novel  problems). 


67 


Conclusions  for  IASs: 

1 .  The  authors  advocate  that  it  is  imperative  to  understand  the  operator’s  role  within  the  system 
to  determine  the  appropriate  system  support  for  that  role.  The  analysis  of  the  operator’s  role 
can  guide  the  system  design. 

2.  A  full  associate  system  is  more  technically  difficult  to  implement  (e.g.,  PA  project)  because  the 
system  is  expected  to  perform  at  the  same  level  of  abstraction  as  the  human  operator. 

3.  Intelligent  aiding  systems  (e.g.,  full  associate  system)  should  provide  assistance  with  the  basic 
functions  of  assessment,  planning,  co-ordinating  and  acting  (to  mimic  human  information 
processing  and  problem-solving  abilities). 

4.  Functional  architectures  are  a  good  way  to  implement  lAHs  that  support  strong  interactions 
and  tight  integration.  That  is,  the  behaviours  required  by  the  domains  (e.g.,  tasks)  are  shared 
between  the  system  and  the  human  across  the  functional  components. 

5.  In  order  to  support  an  associate  relationship  with  the  system,  the  authors  claim  that  function 
allocation  should  be  flexible  and  dynamic,  driven  more  by  the  situation  and  context,  than  by  the 
preservation  of  a  sole  sources  of  control  authority  (unlike  the  CAMMY  and  CE  project  that  are 
driven  by  pilot  control). 

6.  Operator  trust  is  enhanced  by  IAH  system  consistency  and  correctness  (e.g.,  decisions  and 
actions  are  consistent  and  predictable). 

7.  The  plan-goal  graph  (PGG)  modelling  approach  was  developed  to  address  the  problem  of 
intent  referencing  and  used  in  the  HEC  model  as  a  means  to  predict  pilot  intent.  Intent 
recognition  is  achieved  by  differentiating  the  goals  from  the  behaviour  of  the  operator. 

8.  Operator  errors  with  increased  risk  of  severe  consequences  (especially  without  corrective 
action)  should  require  assertive  intervention  and  action  aiding  by  the  system  (e.g.,  auto-pilot  is 
automatically  turned  on  when  the  pilot  loses  consciousness). 

9.  System  transparency  is  needed  to  maintain  awareness  of  system  functioning  (system  state; 
e.g.,  when  automation  is  turned  on  or  off)  and  a  sense  of  operator  control. 

10.  The  system  should  conform  to  the  pilots’  mental  model.  A  mental  model  is  a  representation 
formed  by  an  operator  of  a  system  and/or  task  and  is  based  on  previous  experience  and 
current  observation.  This  provides  a  basis  for  the  operators  understanding  of  system 
functionality  which  can  influence  their  performance  on  tasks. 


Reference: 


Taylor,  R.  M.  and  Finnie,  S.  E.  (1997).  The  Cognitive  Cockpit:  Adaptation  and  control  of  complex 
systems.  Proceedings  of  the  4th  Electronic  Crew  Conference,  Kreuth,  Germany,  23-26  September 
1997. 


Overview: 

Considers  the  relevance  of  control  theory  as  a  broad  framework  for  Human-Electronic  Crew 
teamwork  in  dealing  with  uncertainty  in  dynamic  situations.  It  examines  possible  "Cognitive 
Cockpit"  architectures  for  future  HEC  systems,  which  adapt  to  both  control  feedback  and  feed 
forward  requirements,  encompassing  the  uncertainty  in  dynamic  situations. 

•  The  authors  found  that  control  problems  arising  from  poor  mode  awareness  are  commonly 
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reported  with  complex  automated  systems. 

•  There  is  evidence  that  compliance  (i.e.,  over-reliance)  to  automation  can  occur  if  the 
automation  is  not  clear  when  and  how  it  is  applied  by  the  IAH  system  (i.e.,  the  mode  of  system 
state  is  not  transparent). 

•  The  aim  of  the  COGPIT  work  is  to  find  appropriate  blend  of  feed  forward  and  feedback 
information  for  supporting  the  intentions  of  commanders  and  operators  that  enable  them  to  be 
in  control  of,  rather  than  controlled  by,  the  system. 


Conclusions  for  IASs: 

1 .  The  authors  have  concluded  that  the  operator  should  have  a  means  of  monitoring  system 
functioning  (e.g.,  checking  and  direction),  especially  in  uncertain  and  highly-complex 
environments. 

2.  The  operator  should  be  actively  involved  in  generating  plans  and  determining  execution 
triggers  of  automation. 

3.  Automated  support  (allocation  of  tasks  to  the  system)  can  be  structured  according 
Rasmussen’s  Skills,  Rule  and  Knowledge  Framework  (cognitive  systems  engineering).  It  was 
found  that  this  approach  can  highlight  opportunities  for  the  dynamic  allocation  of  functions  to 
the  system  (i.e.,  initiation  of  automation). 

4.  System  feed-forward  and  feedback  information  can  be  provided  to  the  operator  based  on  the 
perceptual  control  model  (i.e.  IMPACT).  That  is,  intelligent  aiding  should  be  designed  to 
support  pilot  desires  or  intentionality  (i.e.,  goal-based  support). 


3.4.2  Associate  Systems  Technology 

A  major  hurdle  in  the  development  of  IAHs  is  the  requirement  for  accurate  monitoring  of  the 
physical  and  cognitive  state  of  the  operator.  Such  monitoring  is  required  to  ensure  that  the 
autonomous  changes  in  automation  level  or  OMI  adaptation  being  executed  by  the  system  are 
appropriate  to  maintain  optimised  operator  performance  and  information  processing. 

Although  a  number  of  solutions  are  currently  in  development,  there  are  currently  no  mature 
means  of  producing  such  monitoring.  Further  progress  in  the  field  of  operator  state  monitoring 
will  be  required  if  IAHs  are  to  be  realised  in  war-fighting  domains. 

Due  to  the  difficulty  in  realising  effective  adaptation,  developers  have  attempted  to  avoid  the 
requirement  for  adaptation  based  on  the  physical  and  cognitive  state  of  the  operator  (although 
some  small  elements  of  adaptation  may  exist  where  achievable)  and  instead  focused  on 
providing  mission  planning  and  decision  making  support  in  the  form  of  co-operative, 
intelligent  sub-systems  acting  in  a  team-based  manner  with  the  operator.  These  developments 
have  typically  been  characterised  as  Associate  Systems  Technology  (AST),  exemplified  by 
the  US  Pilot’s  Associate  and  Rotorcraft  Pilot’s  Associate  programmes. 

Associate  System  Technology,  otherwise  known  as  integrated  real-time  intelligent  systems, 
provides  intelligent  decision  aiding.  The  intention  is  not  to  replace  aircrew,  but  to  organise 
uncertain  or  conflicting  information  through  data  fusion  and  to  provide  data  interpretation, 
hypothesis  formulation,  planning  and  decision  aiding  to  the  operator.  In  other  words,  the 
system  is  not  making  decisions;  instead  it  is  doing  some  of  the  ground-work  so  that  the 
operator  can  choose  the  best  solution  in  any  given  situation. 
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An  associate  system  is  characterised  by  several  attributes.  An  associate  knows  how  to  help 
without  being  told  by  the  operator.  An  associate  shares  a  sense  of  purpose  with  the  operator 
(i.e.,  common  intent)  and  formulates  plans  to  achieve  the  joint  purpose.  An  associate  actively 
assesses  the  world  and  communicates  important  aspects  of  the  situation  and  response  options 
to  the  operator.  An  associate  actively  aids  task  execution  as  authorised  by  the  operator.  It  can 
contribute  greatly  to  the  operator’s  ability  to  see  and  comprehend  the  battlefield  in  all 
conditions,  a  rapidly  collect,  synthesise,  and  disseminate  battlefield  information,  and  take 
immediate  and  effective  actions. 

Associate  Systems  typically  have  two  components: 

•  Cognitive  Decision  Aiding.  The  Cognitive  Decision  Aiding  Sub-system  (CDAS) 
performs  situation  assessment,  planning,  and  cockpit  information  management. 
Cognitive  decision  aiding  is  applied  at  the  crew  level  to  augment  the  crew’s  decision 
making  process  (i.e.,  an  en-route  mission  change  or  actions-on-contact  with  the 
enemy).  Unique  crew  aiding  behaviours  partition  the  tasks  between  the  crew  and  the 
CDAS  so  to  best  utilise  the  advantages  of  the  human  crew  member  and  the  computer 
planner  assessor.  Crew  selectable  levels  of  the  associate  functioning  are  also  possible. 

•  Crew  Intent  Estimation.  The  purpose  of  Crew  Intent  Estimation  (CIE)  is  to  ensure  that 
the  operator  is  always  in  charge  of  the  associate  system’s  functioning,  so  that  the 
system  is  never  pursuing  old  or  counter-productive  behaviours.  It  tracks  operator 
behaviour  and  provides  co-ordinating  information  to  planners  and  assessors 
attempting  to  support  the  operator.  For  example,  there  are  specific  mechanisms  for 
resolving  actions  that  can  have  multiple  interpretations,  miscues  and  confusions 
between  what  the  associate  thinks  the  operator  is  doing  versus  what  the  CIE  just 
determined. 

3.4.2. 1  Pilot's  Associate  (United  States) 

The  Pilot’s  Associate  program  was  part  of  a  $40  million  effort  by  Lockheed  (1986-1992)  to 
mobilise  artificial  intelligence  technology  for  real  world  operational  problems  in  the  United 
States  Department  of  Defence.  The  Pilot’s  Associate  was  to  be  an  “electronic  back-seater”,  an 
aid  to  the  pilot  that  could  monitor  the  current  external  situation,  assess  threats,  and  plan 
reactions  to  events,  thus  demonstrating  the  effectiveness  of  integrated,  real-time  intelligent 
systems  in  the  air  combat  domain.  This  project  provided  the  first  associate  system  capable  of 
operating  in  close  co-operation  with  its  human  counterpart.  Although  the  Pilot’s  Associate 
program  ended  in  1992,  it  has  had  a  significant  impact  (e.g.,  parts  of  the  Pilot’s  Associate 
have  been  incorporated  into  the  avionics  of  the  F-22). 

Between  1990  andl992,  Honeywell  developed  the  Learning  System  Pilot  Aiding  (LSPA) 
project.  The  goal  of  the  LSPA  project  was  to  demonstrate  that  explanation-based  learning,  a 
type  of  machine  learning  that  utilizes  a  worked  example  of  a  problem  as  a  problem-solving 
method,  would  be  effective  for  automatically  generating  knowledge  bases  for  Pilot’s 
Associate  from  single  instances  of  observed  pilot  behaviour.  At  its  conclusion,  LSPA  showed 
successfully  that  finished  knowledge  bases  for  tactical  planning  and  for  information 
management  could  be  created  directly  from  time  histories  of  pilot  behaviour  in  a  simulator. 
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3.4.2.2  Rotorcraft  Pilot's  Associate  (United  States) 


The  Rotorcraft  Pilot’s  Associate  advanced  technology  demonstration  programme  started  in 
June  1993  with  conceptual  design  studies.  Manned  simulation  and  flight  test  evaluations  were 
completed  in  1998.  The  objective  of  the  RPA  programme  was  to: 

•  Apply  AI  and  state-of-the-art  computing  technologies  to  manage  and  integrate  next 
generation  mission  equipment  and  battlefield  information;  and, 

•  Enhance  the  lethality,  survivability,  and  mission  effectiveness  of  combat  helicopters. 

The  RPA  technology  demonstration  was  intended  to  enhance  crew  performance  and  mission 
execution  by  applying  associate  system  technology  to  the  cockpit  to  intelligently  manage  the 
sub-systems  as  well  as  the  information  flow  to  and  from  the  crew.  Operationally,  the  RPA 
aims  to  free  the  crew  to  execute  their  mission  more  effectively  by  managing  the  myriad  of 
significant  and  insignificant  data  from  the  sensors.  RPA  attempts  to  anticipate  crew  needs 
based  on  mission  context  and  provide  the  right  information  at  the  right  time,  and  when 
authorised,  will  take  the  initiative  to  actively  aid  the  crew  and  execute  tasks. 

Functionally,  RPA  consists  of  a  Cognitive  Decision  Aiding  System  (CDAS)  and  data  fusion. 
CDAS  has  three  basic  interdependent  modules:  internal  and  external  situation  assessors; 
planning  modules;  and  a  cockpit  information  manager.  The  core  architecture  contains  CDAS, 
controls  and  displays  processors  for  each  crew-station,  interfaces  to  a  suite  of  advanced 
mission  equipment  sub-systems,  a  data  distribution  system,  and  data  fusion.  The  output  of 
data  fusion  is  used  to  assess  the  battlefield  in  terms  of  threats,  targets,  obstacles,  and  friendly 
troops.  Entity  groups  and  relationships  are  identified  in  order  to  recognise  events  that  pose 
near  term  danger.  Threat  capabilities  and  intentions  are  estimated  and  the  possible  impacts  on 
mission  plans  are  considered.  Information  is  disseminated  to  the  team,  Tactical  Operations 
Centre,  and  intelligence  networks  as  appropriate.  CDAS  prioritises  and  co-ordinates  target 
attack,  predicts  success  probability  considering  team  co-ordination,  and  monitors  progress. 

In  addition,  six  onboard  planners  use  the  information  produced  by  data  fusion  and  assessment 
sub-systems  to  aid  the  crew  in  reacting  quickly  and  efficiently  to  changing  and  unexpected 
events  that  occur  during  the  mission.  These  planners  are: 

•  Survivability  planner.  Continuously  updates  countermeasures  and  evasive  manoeuvre 
recommendations.  Intelligently  configures  mission  equipment  such  as  jammers  or 
chaff  dispensers  and  fine-tunes  flight  envelopes  to  minimise  signature  to  known  and 
probable  threats; 

•  Sensor  planner.  Contextually  optimises  sensor  settings  such  as  field-of-view  or 
frequency  band  selection,  employs  multiple  sensors  synergistically,  co-ordinates  the 
team’s  use  of  sensors  by  allocating  coverage  areas  and  performing  multi-ship  data 
correlation,  and  compensates  for  sensor  equipment  failure  and  degradation; 

•  Communications  planner.  Automates  routine  calls,  radio  selection  and  keying  in 
compliance  with  standard  operating  procedures.  It  adapts  for  communication 
equipment  failures,  current  environmental  conditions,  and  changing  sector 
boundaries.  Also  recommends  the  optimum  time  and  location  for  calls,  estimates  the 
probability  of  reception  and  intercept  and  co-ordinates  team  radio  calls; 
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•  Attack  planner.  Selects  priority  fire  zones  and  no-fire  zones,  co-ordinates  target  scans 
and  target  designations,  recommends  appropriate  weapons  using  onboard  weapons 
models,  automates  assistance  for  running  fires,  recommends  engagement  areas  and 
battle  positions,  and  co-ordinates  and  synchronises  the  attack  to  minimise  enemy 
response; 

•  Reconnaissance  planner.  Develops  a  co-ordinated,  multi-platform  effort  to  safely  and 
efficiently  perform  zone,  route  or  area  reconnaissance.  Also  maximises  the 
information  collected  and  minimises  the  likelihood  of  ownship  detection;  and, 

•  Route  planner.  Re -plans  flight  routes  over  large  areas  with  high  resolution.  Factors 
such  as  distance,  time,  fuel,  flight  regime,  and  known  probable  moving  and  stationary 
threats  are  considered. 


Reference: 


Miller,  C.A.  and  Dorneich,  M.C.  (2006).  From  Associate  Systems  to  Augmented  Cognition  25 
Years  of  User  Adaptation  in  High  Criticality  Systems.  Poster  presented  at  the  Augmented 
Cognition  conference,  October  2006,  San  Francisco. 


Overview: 

In  the  1980’s,  the  U.S.  Air  Force  initiated  the  development  of  a 
human-adaptive,  information,  and  automation  management 
technology  known  as  the  “Pilot’s  Associate”. 

What  is  it?  PA,  and  all  of  the  subsequent  associate  systems, 
consisted  of  an  integrated  suite  of  intelligent  subsystems  that 
were  designed  to  share  (among  themselves  and  with  the  pilot) 
a  common  understanding  of  the  mission,  the  current  state  of 
the  world,  the  aircraft  and  the  pilot.  Associate  systems  were 
designed  to  use  the  shared  knowledge  to  plan  and  suggest  courses  of  action,  and  to  adapt  cockpit 
information  displays  and  the  behaviour  of  aircraft  automation. 

Automation  of  tasks:  Tasks  are  automated  only  in  line  with  the  operator’s  goals  and,  whenever 
feasible,  to  be  authorized  by  the  operator.  Operator  control  of  the  automation  was  established 
either  during  the  mission,  immediately  prior  to  execution  of  automation,  or  pre-mission,  in  a  pre¬ 
authorization  mode. 

Lessons  Learned  from  PA  efforts:  Associate  systems  were  and  are  the  predecessors  of 
augmented  cognition  (AugCog)  technologies.  While  there  are  many  similarities  between  PA  and 
AugCog  systems,  there  are  also  some  many  differences: 

•  Associate  systems  leave  the  pilot  “in  charge”  which  is  extremely  important  in  high  criticality 
domains.  To  increase  the  chance  of  operator  acceptance,  it  is  important  to  consider  that  the 
operator  should  be  kept  in  the  loop.  The  authors  claim  that  if  the  pilot  is  responsible  for  the 
actions  of  the  aircraft,  then  the  pilot  must  be  the  final  authority  of  the  aircraft’s  actions. 

•  All  components  (e.g.,  sensors,  information  fusion  technologies,  and  interfaces)  should  be  co¬ 
developed  and  evaluated  in  concert. 

•  A  task-based  framework  was  an  effective  way  to  coordinate  a  variety  of  processes  (or 
subsystems)  and  minimize  the  costs  of  revising  or  extending  them. 

•  Personification  (or  customization)  is  out  of  place  in  high  criticality  domains  (and  possibly  other 
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domains,  as  HCI). 

•  OMI  design  proved  to  be  an  important  component  of  the  associate  system.  It  was  often  found 
that  the  OMI  will  highlight  anything  that  is  wrong  with  any  module,  and  errors  in  the  design  of 
the  OMI  will  make  all  other  aspects  of  the  associate  less  effective. 

•  Adaptation  of  systems  to  individual  differences  and  operator  expectations  (but  not 
customization)  can  have  large  payoffs  for  fitting  a  system  to  an  operator’s  needs  and 
capabilities. 


Conclusions  for  IASs: 

Several  “Lessons  Learned”  from  the  PA  efforts  were  outlined  in  this  paper,  which  have  implications 

for  the  development  of  IA  systems  (see  Lesson  learned  from  PA  efforts  above): 

1 .  Importance  of  operator  acceptance  and,  therefore,  importance  of  keeping  human  “in  charge”. 
The  authors  advocate  migrating  control  to  a  supervisory  level  (where  the  human  varies  the 
amount  and  level  of  automation)  and  that  the  system  should  not  rely  too  heavily  on  inferred 
operator  state  or  intent.  This  can  increase  human  out-of-the-loop  problems. 

2.  Importance  of  co-development  and  progressive  testing:  Development  efforts  and  individual 
technologies  should  be  co-developed  and  used  in  collaboration,  which  can  aid  the 
development  of  an  overall  system.  For  instance,  the  development  of  neurophysiological 
sensors  or  “meters”,  other  means  of  assessing  operator  state,  and  the  development  of 
methods  for  “augmenting”  cognition  through  information  display  technologies  need  to  be  co¬ 
developed  and  evaluated  in  concert. 

3.  Benefits  of  an  explicit,  integrative  framework  (task  model).  Knowledge  of  the  task  context  can 
help  develop  systems  that  manage  task  demand  and  increase  operator  performance. 

4.  Operator-machine  interactions.  More  effective  means  of  interactions  between  the  operator  and 
the  system  may  be  achieved  if  the  designer  approaches  an  intelligent  system  as  a  “personified 
agent”  whose  goal  it  is  to  aid  the  operator  and  to  recognize  that  the  operator  might  have 
feelings  or  attitudes. 

5.  Importance  of  interface  and  interaction  design.  A  system,  especially  for  a  high  criticality 
domain,  should  be  designed  with  a  system  failure  in  mind.  The  authors  provide  some  methods 
that  can  accomplish  this:  give  the  operator  the  ability  to  override  and  turn  off  the  technology; 
allow  the  operator  to  explicitly  authorize  a  display  modification,  to  be  notified  of  pending 
changes,  to  be  notified  of  executed  changes,  and  to  rapidly  return  to  a  previous  display  state. 

6.  Importance  of  learning,  especially  individuation.  Recording  individual  performance  effects 
could  serve  to  provide  a  powerful  means  of  adapting  system  behaviour  to  the  individual. 


Geddes,  N.D.,  and  Shalin,  V.L.  (1997).  Intelligent  decision  aiding  for  aviation.  Technical  Report 
(No.  ISSN  1402-7585).  Prepared  for  Linkoping  Institute  of  Technology,  Linkpoping,  Sweden. 


Overview: 

Intelligent  decision  aiding  technologies  (adaptive  systems  and  automation)  are  reviewed  in  the 
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context  of  the  aviation  domain.  This  report  explores  issues  of  system  architecture,  development 
and  integration  methods,  and  approaches  to  the  test  and  evaluation  of  large-scale  intelligent  aiding 
systems.  The  focus  is  based  on  the  Pilot’s  Associate  program.  The  following  briefly  describes  the 
development  strategies  used  to  develop  IAI  systems: 

•  Human-Centered  Design  (HMD)  perspective:  This  perspective  determined  that  the  role  of  the 
human  in  the  system  is  based  on  an  operational  philosophy.  It  identified  what  types  of  roles 
humans  should  play  in  the  system  (e.g.,  as  authority  and  agent). 

•  Design  Representations:  The  use  of  Intelligent  Object-Oriented  Design  (IOOD)  is  a  series  of 
steps  that  is  well  suited  to  transform  requirements  (e.g.,  system,  operator,  organization)  to 
more  abstract  views  of  objects  (refer  to  pages  65-68  for  details  on  this  process). 

•  Iterative  Design  Process:  This  process  recognizes  the  need  for  feedback  into  the  design  and 
requirements  process  to  ensure  that  the  design  of  the  system  evolves.  This  process  outlines  a 
series  of  prototyping  cycles  which  is  a  common  means  of  organizing  iterative  development. 

•  Application  of  development  tools:  It  was  found  that  an  iterative  development  is  most  productive 
when  supported  by  a  set  of  management,  design  and  testing  tools  (e.g.,  Plan  Goal  Graph  Tool, 
Display  Analyst) 


Conclusions  for  IASs: 

1 .  An  HMD  process  is  an  effective  approach  to  identify  the  types  of  roles  humans  should  play  in 
the  system  (i.e.,  as  agents  and  authority  over  the  initiation  of  automation). 

2.  The  authors  found  that  knowledge-based  systems,  such  as  intelligent  adaptive  systems, 
require  more  elaboration  at  higher  levels  of  abstraction  (e.g.,  plan-goal  graphs;  abstract 
processes,  behaviours  and  use  case  templates).  This  can  be  achieved  through  IOOD.  Object- 
oriented  languages  are  well  suited  to  transform  requirements  and  support  the  development  of 
IAH  systems. 

3.  An  iterative  design  process  is  a  good  way  to  enable  the  design  and  requirements  process.  This 
approach  ensures  that  the  system  is  verified  against  its  design  and  validated  against  its 
requirements. 

4.  Complex  intelligent  adaptive  systems  require  many  agents  and  frameworks  (i.e.,  operational 
knowledge  representations).  Interaction  protocols  can  be  used  to  ensure  that  the  system 
operates  as  a  whole. 

5.  There  are  now  a  large  number  of  well-developed  reasoning  algorithms  and  operational 
knowledge  representations.  While  a  wealth  of  processes  (i.e.,  algorithms)  and  representations 
is  often  necessary  for  complex  systems,  the  authors  are  careful  to  indicate  that  the  system 
should  take  the  form  that  best  suits  the  process  desired. 

6.  The  authors  have  noted  that  the  development  process  requires  tool  support  (e.g.,  PGG)  but 
that  the  tools  presently  available  (as  of  1997)  fall  short  for  knowledge-based  systems.  System 
designers  must  therefore  anticipate  the  need  to  develop  tools  along  with  the  product. 

7.  By  starting  with  a  core  system  and  an  open  architecture,  IA  systems  can  evolve  into  broader 
and  more  powerful  support  systems.  For  instance,  rather  than  starting  in  the  cockpit  of  a 
helicopter  of  single  jet  seat  fighter,  a  better  starting  place  may  be  within  the  mission  crew. 
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Reference: 


Miller,  C.  and  Hannen,  M.  (1998).  User  acceptance  of  an  intelligent  user  interface:  A  rotorcraft 
pilot's  associate  example.  In  M.  T.  Maybury  (Ed.).  Proceedings  of  the  4th  International  conference 
on  intelligent  user  interfaces  (pp.  109-116).  New  York,  NY:  ACM  Press. 


Overview: 

This  paper  details  the  high  level  architecture  of  the  Cockpit  Information  Manager  (of  the  RPA).  It 

emphasizes  how  pilot  behaviours  are  monitored,  crew  intent  is  estimated,  symbols  are  selected 

and  de-cluttered,  windows  are  located,  and  the  automated  pan  and  zoom,  and  allocation  of  tasks 

are  implemented. 

Cockpit  Information  Manager  (CIM)  Architecture 

•  The  Ul  is  primarily  task-based. 

•  This  module  is  responsible  for  determining  the  current  and  near-future  tasks  of  the  crew  and 
then  adjusting  the  cockpit  configuration  to  meet  task  needs. 

•  Tasks  are  allocated  to  agents  (human  and  machine)  through  this  interface  through  functions 
that  are  required  for  specific  tasks  (full  associate  relationship). 

•  Task  and  context  information  are  provided  to  the  CIM  by  the  Task  Network  and  Context  Model. 

•  A  “Crew  Intent  Estimator”  is  used  to  interpret  pilot  actions  and  world  events  against  mission 
plans  in  the  “Task  Network”,  which  determines  whether  the  pilots  are  following  mission  model 
or  are  attempting  alternate  plans  or  goals  (this  is  similar  to  the  DIAMAnD,  DRDC  and 
Intelligent  Classroom  frameworks  that  use  plan  generation  and  recognition). 

•  CIM  maintains  an  operator  profile  (e.g.,  violations  in  pilot  expectations,  information  needs,  etc). 

•  CIM  performs  six  primary  activities  observable  by  the  pilot:  intent  estimation,  task  allocation, 
page  selection,  symbol  selection/de-clutter,  window  placement  and  pan  and  zoom. 


Conclusions  for  IASs: 

1 .  A  full  associate  relationship  approach  is  taken  by  the  RPA  program  for  providing  system 
assistance. 

2.  This  approach  requires  that  pilot  intent  and  error  recognition  are  monitored  and  estimated  to 
provide  context  appropriate  assistance  (this  is  done  with  a  Crew  Intent  Estimator). 

3.  A  World  State  or  “Context”  Model  is  used  to  determine  context. 

4.  A  task  and  goal-based  approach  in  the  form  of  a  Task  Network  is  used  to  estimate  pilot  intent 
and  error  recognition  with  pre-define  tasks/goals. 

5.  An  operator  profile  is  used  to  determine  pilot  expectations,  their  information  needs,  etc. 

6.  Preliminary  results  from  a  simulation  test  found  that  the  CIM  behaviours  are  contributing  to 
perceived  pilot  effectiveness,  reducing  workload  and  are  gaining  pilot  acceptance. 
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3.4.3  Cockpit  Assistant  System  (CASSY)  (Germany) 


Reference: 


Gerlach,  M.  and  Onken,  R.  (1995).  CASSY-  The  electronic  part  of  the  human-electronic  crew. 
Proceedings  of  the  3rd  international  workshop  on  human-computer  teamwork  (Human-Electronic 
Crew:  Can  we  trust  the  team?).  Cambridge,  UK,  27-30  September  1994. 


Overview: 

The  knowledge-based  commercial  aircraft  Cockpit  Assistant 
System  is  a  civil  aviation  cockpit  assistant  project  developed 
as  an  intelligent  decision  aid.  It  emphasises  pilot  assistance 
through  situation  assessment  and  re-planning  in  flight. 

Situation-dependent  assistance  with  flight  planning  is  guided 
by  a  normative  pilot  model,  goal  conflict,  pilot  intent,  and  error 
recognition  functions.  It  also  aids  in  the  execution  of  pilot 
selected  functions. 

CASSY  is  composed  of  several  situation  assessment  modules  that  interface  with  the  flight  crew, 
the  aircraft,  and  air  traffic  control  (ATC),  which  all  collaborate  with  each  other: 

•  Automatic  Flight  Planning.  This  module  generates  a  complete  global  flight  plan  based  on  its 
knowledge  of  mission  goal,  ATC  instructions,  aircraft  systems  status  and  environmental  data. 
The  flight  plan(s)  is  presented  as  a  proposal  which  the  crew  accepts  or  modifies,  and  once 
chosen,  serves  as  a  knowledge  source  for  other  CASSY  modules. 

•  Piloting  Expert.  Uses  the  valid  flight  plan  and  processes  the  crew  model  to  generate  necessary 
crew  actions. 

•  Pilot  Intent  and  Error  Recognition.  The  expected  crew  actions  are  compared  with  the  shown 
behaviour  of  the  crew.  Crew  actions  are  derived  indirectly  by  interpreting  aircraft  data.  If  given 
tolerances  are  violated,  the  crew  will  be  informed  by  hints  and  warnings  and  the  detected 
mistake  is  alerted  to  the  pilots. 

•  Monitoring.  Enable  the  system  to  recognize  and  interpret  current  situations  (flight  status, 
environment,  and  systems). 

•  Execution  Aid.  Automation  of  tasks  which  are  controlled  by  the  crew. 

The  Dialogue  Manager:  This  is  the  OMI  that  enables  the  operator  to  interact  with  the  system  via  a 
graphic/alphanumeric  display  and  speech  recognition. 

This  paper  describes  the  results  of  the  CASSY  flight  tests.  The  flight  tests  evaluated  flight 
management  KBS  for  rerouting  of  civil  aircraft.  Results  of  flight  test  were: 

•  System  performance:  Correct  expected  pilot  actions  were  generated  and  pilot  errors  were 
detected  and  appropriate  warnings  issued;  and 

•  Operator  acceptance:  CASSY  was  well  accepted;  autonomous  flight  plan  appreciated,  warning 
and  hints  considered  justified. 

The  CASSY  project  is  a  successful  real-time  demonstration  of  an  intelligent  adaptive  system 
implemented  in  a  real  and  not  virtual  environment.  This  project  led  to  the  CAMMA  military  cockpit 
assistant  project. 


ROLE 

AGENT 

AUTHORITY 

ACQUIRE 

a 

a  © 

ANALYZE/ 

PRESENT 

a 

a  © 

DECIDE 

a 

a  © 

ACT 

a  © 

© 

76 


Conclusions  for  IASs: 

1 .  Flight  tests  proved  that  intelligent  decision  aiding  is  feasibly  possible  and  well  accepted  by 
operators. 

2.  The  CASSY  project  is  a  successful  real-time  demonstration  of  an  intelligent  adaptive  system 
implemented  in  a  real  and  not  virtual  environment. 

3.  Situation  assessment  is  an  important  feature  of  a  successful  intelligent  system. 


Reference: 


Wittig,  T.  and  Onken,  R.  (1992).  Pilot  intent  and  error  recognition  as  part  of  a  knowledge  based 
cockpit  assistant.  Proceedings  of  the  AGARD  conference  on  Combat  automation  for  airborne 
weapon  systems:  Man/machine  interface  trends  and  technologies,  Edinburgh,  UK,  19-22  October 
1992. 


Overview: 

This  paper  describes  the  concept  and  functionality  of  the  Cockpit  Assistant  System;  including  pilot 
intent  and  error  recognition.  Evaluation  of  CASSY  in  a  flight  simulator  is  also  described. 

The  Pilot  Intent  and  Error  Recognition  (PIER)  module  supports  the  pilot  crew  with  regard  to  the 
monitoring  and  planning  task,  and  provides  assistance  for  a  number  of  plan  execution  functions. 
During  the  whole  flight,  the  module  monitors  pilot  activities  and  the  flight  status  in  order  to  detect 
deviations  from  the  actual  flight  plan  immediately.  The  current  flight  situation  is  evaluated,  and  pilot 
behaviour  is  analyzed  over  a  certain  period  of  time  and  through  either  pilot  intent  or  error.  Pilot 
errors  lead  to  warning  messages  and  modifications  of  the  flight  plan.  The  module  is  mainly 
performed  by  use  of  an  “inference  algorithm”  based  on  known  intent  hypotheses. 

PIER  is  composed  of  the  following  structure:  situation,  pilot  behaviour  and  pilot  situation 
interpretation;  determinant  of  pilot  action  sequence,  classification  of  crew  intent;  final  intent  and 
error  inference. 


Conclusions  for  IASs: 

1 .  Operator  intent  and  error  recognition  can  be  an  effective  means  of  providing  adaptive 
assistance. 

2.  Uncertainties  can  be  evaluated  using  certainty  factors  (probabilistic  reasoning  such  as  Bayes’ 
Theorem). 

3.  Algorithms  based  on  a-priori  probabilities  for  possible  hypotheses  have  proven  useful  for 
recognizing  and  estimating  operator  intent.  The  probabilities  can  be  modified  with  respect  to 
operator  actions. 
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3.4.4  DRDC-Toronto  IAI  UAV  Project  (Canada) 


Hou,  M.  Kobierski,  R.,  and  Herdman,  C.  (2006)  Design  and  Evaluation  of  Intelligent  Adaptive 
Operator  Interfaces  for  the  Control  of  Multiple  UAVs.  Proceedings  of  the  RTO  Human  Factors  and 
Medicine  Panel  Symposium.  Biaritz,  France. 


Overview: 

This  paper  reports  on  a  multi-phase  project  to  investigate  the 
potential  of  artificial  intelligence  for  the  control  of  multiple 
UAVs.  These  three  phases  include  IAI  concept  development, 
interface  prototyping,  and  experimentation.  Human-in-the-loop 
trials  in  a  realistic  mission  scenario  were  conducted  to 
examine  the  performance  model  developed  by  DRDC. 

Experimentation:  Two  modes  were  investigated.  The  first 
mode  required  operators  to  use  a  conventional  interface  to 
control  the  UAVs  and  the  second  mode  included  interface  automation  that  used  an  IAI.  The 
difference  between  mission  activities  with  and  without  automation  was  examined. 

IAI  Experimental  Environment:  The  trails  were  conducted  in  a  synthetic  environment.  Three  control 
consoles,  consistent  with  the  UAV  crew  position,  were  setup  to  replicate  CP140  tactical 
compartment  multifunction  workstations.  Workstations  were  designed  to  communicate  with  virtual 
UAVs  through  fully  functional  real  world  software  interfaces.  Each  member  of  the  UAV  crew  was 
provided  with  their  own  workstation,  which  consisted  of  a  main  display  screen,  a  keyboard,  a 
programmable  entry  panel,  a  trackball/mouse,  and  a  joystick. 

The  UAV  sensor  operator’s  primary  display  was  designed  for  providing  the  information  necessary 
to  manage  and  extract  information  from  a  large  number  of  sensors.  The  main  display  area  was 
highly  customizable,  allowing  the  operator  the  flexibility  in  creating  layouts  for  the  display  of  sensor 
data  and  in  switching  between  those  layouts. 

The  IAI  agents  were  designed  to  follow  a  defined  sequence  as  follows: 

Step  1 :  Gather  status  information  about  all  active  UAVs,  the  tracks  pertinent  to  those  UAVs  and 
the  current  display  configuration. 

Step  2:  Analyse  the  information  with  respect  to  pre-defined  rules  and  determine  which  events  have 
occurred. 

Step  3:  Prioritize  the  events  with  respect  to  pre-defined  prioritization  rules. 

Step  4:  Execute  pre-defined  tasks  for  each  identified  event  following  the  order  of  prioritization. 

Six  software  agents  were  designed  and  implemented  to  provide  the  following  adaptive  aiding: 
route  planning,  route  following,  screen  management,  inter-crew  communication,  sensor 
management,  and  data  link  monitoring. 

Results.  Results  indicate  that  operators  performed  more  effectively  when  the  IAI  multi-agent 
system  was  selected  ON.  When  IAI  was  ON,  completion  time  for  critical  task  sequences  were 
shortened,  less  tasks  were  shed,  the  UAV  trajectory  scores  were  better  and  much  less  time  was 
spent  in  the  no-fly  areas.  In  addition,  operators’  overall  SA  was  improved  and  overall  workload  was 
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reduced  as  well. 

Several  recommendations  are  provided  for  the  design  and  implementation  of  lAls,  as  outlined 
below. 


Conclusions  for  IASs: 

1 .  Authors  suggest  that  a  more  complete  set  of  agents  would  result  in  significantly  increased 
crew  performance.  The  authors  felt  that  the  actual  number  of  agents  incorporated  in  the 
experiment  was  limited,  and  the  quality  of  the  implementation  was  fair-to-good  though  not  at 
the  quality  of  a  production  system.  This  suggestion  should  be  taken  in  light  of  other  research 
on  adaptive  aiding. 

2.  Results  suggest  that  Hierarchical  Goal  Analysis  (HGA)  was  an  effective  means  of  analysis  for 
supporting  in  route  planning,  route  following,  and  inter-crew  communication. 

3.  Results  suggest  that  operators  of  intelligent  adaptive  interfaces  should  be  given  a  training 
period  before  actually  using  the  system,  particularly  in  life-critical,  mission-critical  systems. 

4.  A  hybrid  IAI  based  on  experience  with  the  adaptive  system  may  increase  the  operator’s 
understanding  of  the  system  and  its  impact,  a  phase  dependent  mix  between  fully  automatic 
and  operator-controlled  adaptation. 

5.  The  system  should  inform  the  operator  of  interface  changes.  For  instance,  the  IAI  should  either 
indicate  for  a  few  seconds  where  it  is  going,  or  indicate  what  has  changed. 

6.  The  interface  should  allow  the  operator  to  return  to  the  system  state  that  was  in  effect  before 
the  IAI  reconfigured  the  display  to  increase  the  sense  of  operator  control. 

7.  The  design  of  each  intelligent  agent  in  a  rapid  prototype  operator  interface  should  be  based  on 
reality. 

8.  Intelligent  agents  should  be  made  aware  of  the  world  state  by  accessing  data  fusion  interim 
variables  and  associated  probabilities.  The  authors  suggest  that  this  would  allow  the  IAI  to 
produce  strategies  that  “play  the  odds”. 

9.  All  IAI  functions  which  are  studied  during  design  and  development,  and  which  are  incorporated 
into  an  operator  evaluation,  should  be  thoroughly  researched  to  confirm  that  the  concept 
implemented  is  adequate  for  the  assessment. 


Edwards,  J.L.  (2004).  Generic  Agent-  Based  Framework  For  UAV/UCAV  -  Final  Report  -A 
Technical  Report  prepared  by  DRDC  Toronto  (Report  No.  AIMDC  (AC261 ,  February,  2004)). 


Overview: 

A  generic  framework  for  the  design  and  implementation  of  I A  agent-based  systems  is  proposed. 
The  framework  was  constructed  from  the  elements  of  the  following  design  approaches: 
CommonKADS,  MAS-CommonKADS,  IDEF  Standards,  Explicit  Models  Design,  Perceptual  Control 
Theory  and  Ecological  Interface  Design.  Edwards  then  presents  an  overview  of  the  procedure  for 
designing  and  implementing  a  knowledge  based  system  within  the  generic  framework. 
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Framework  Process: 

The  following  steps  were  used  to  generate  the  proposed  framework. 

1 .  As  a  first  step,  CommonKADS  was  used  to  provide  a  set  of  guidelines  for  developing 
knowledge-based  systems.  Six  models  and  their  elements  are  identified  in  this  process: 

•  Organizational  Model  (organizational  or  business  processes); 

•  Task  Model  (high-level  tasks  and  goals  of  agents  in  the  system); 

•  Agent  Model  (who  does  what); 

•  Knowledge  Model  (detailed  knowledge  required  to  perform  the  tasks  that  the  system  will  be 
performing); 

•  Communication  Model  (communication  that  must  occur  among  agents  (human-agent;  agent- 
human)  in  the  knowledge  system);  and 

•  Design  Model  (examines  hardware  and  software  issues  related  to  the  construction  of  the 
knowledge  system.  The  aim  is  to  take  the  implementation-independent  specifications  from  the 
Knowledge  and  Communication  Models  and  develop  a  detailed  design  for  constructing  the 
software  application,  and  in  the  process,  preserve  the  structure  of  those  models.) 

2.  Next,  IDEF  standards  (IDEF3  and  IDEF5)  were  used  to  complement  CommonKADS  in  two 
ways.  IDEF3  can  be  used  to  model  the  processes  and  associated  agents  in  real-time.  IDEF5  is 
used  to  develop  ontologies  (i.e.,  expert  knowledge  elicitation).  For  example,  the  knowledge  elicited 
with  CommonKADS  can  be  used  to  develop  graphical  and  textual  representation  of  language. 
CommonKADS  meanwhile  can  use  to  turn  this  modeling  of  knowledge  into  the  design  of  software 
modules  and  platform  requirements. 

Explicit  Models  Design  (EMD)  was  then  used  to  turn  the  knowledge  derived  from  CommonKADS 
and  IDEF  into  actual  software  modules. 

1 .  Task  Model.  Knowledge  about  tasks  being  performed  (based  on  PCT-based 
hierarchical  goal  analysis); 

2.  User  Model.  Comprised  of  knowledge  relating  to  the  operator’s  abilities,  needs  and 
preferences  (more  like  customization/personalization); 

3.  System  Model.  System’s  knowledge  about  itself  and  its  abilities  (goal  hierarchy  for  each 
agent,  and  automation  level  is  based  on  PACT  from  COGPIT); 

4.  Dialogue  Model.  Knowledge  related  to  communication  among  human  and  software 
agents 

5.  World  Model.  Representing  knowledge  of  the  world  relevant  to  the  purpose  of  the 
software  (i.e.,  aircraft  parameters,  GIS,  human  behaviour,  mission  scenario  and  rules  of 
engagement). 

Plan  recognition  (awareness  of  what  the  operator  is  trying  to  accomplish)  and  plan  generation 
(strategies  for  the  system  to  help  the  operator  accomplish  its  goals)  are  used  to  determine  when 
the  system  should  initiate  automation  of  tasks.  Both  plan  generation  and  recognition  and  feedback 
concepts  are  used  to  support  the  integration  of  PCT  and  EMD  components  into  the  generic 
framework. 

3.  Perceptual  Control  Theory  (PCT)  and  EMD. PCT  and  EMD  are  used  to  turn  the  knowledge 
frameworks  developed  by  IDEF  and  CommonKADS  into  design  techniques  for  how  the  system  will 
function.  For  instance,  how  are  the  goals  an  operator  is  trying  to  achieve  determined,  what  are  the 
plans  for  achieving  those  goals,  and  how  it  can  it  assist  the  operator  most  effectively? 

Goal-driven  control  loop.  PCT  is  used  as  a  framework  to  combine  all  the  modules  and  determine 
when  and  how  the  system  should  provide  assistance  (automation  of  tasks  and  rule-based 
adaptation).  There  are  two  main  concepts  in  PCT. 
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1 .  HGA  of  Tasks  is  used  in  this  process. 

2.  Machine-Learning  techniques:  the  system  modifies  its  adaptation  rules  (reorganizes 
the  control  loops)  depending  on  operator  behaviour  with  the  system  (randomness). 

Automation  of  Tasks.  PCT  is  used  to  develop  control  loop  hierarchies  and  associated  goals  and 
sub  goals.  Together,  they  can  identify  when  the  system  should  provide  assistance,  what  kind  of 
assistance  should  be  given,  and  whether  or  not  it  should  be  initiated  automatically  or  by  operator 
control.  Plan  recognition  is  used  here  (error  recognition  or  deviation  from  plan). 

4.  Software  Agent  Paradigm.  Edwards  takes  a  software  agent-based  approach  to  implementing 
the  framework.  That  is,  how  the  system  provides  dynamic  assistance  is  completely  driven  by 
algorithms.  Edwards  identifies  several  advantages  of  using  an  agent-based  approach.  “That 
“intelligent  autonomous  agents  are  ideally  suited  to  taking  over  some  tasks  for  human  operators, 
serving  to  support  the  goals  of  reduced  manning  and  enhanced  performance  in  a  complex 
environment.  Also,  the  redundancy  and  distribution  across  systems  leads  to  improved  reliability 
and  safety  because,  in  the  event  of  communication  breakdown,  mechanisms  are  in  place  to 
compensate”. 

Edwards  then  details  how  CommonKADS,  IDEF  standards,  EMD,  PCT  and  PACT  can  be  used  to 
develop  what  agents  are  needed  and  how  they  should  be  structured. 

5.  Ecological  Interface  Design  (EID)  is  used  to  help  with  the  development  of  the  Interface,  and  only 
with  the  interface. 

Proposed  DRDC  Generic  Framework 

The  proposed  generic  framework  that  was  constructed  from  the  elements  of  the  following  design 
approaches  is  outlined  below: 

1 .  Construct  the  Organisation  Model  to  describe  the  command  and  control  structure  within 
which  the  project  will  be  developed; 

2.  Construct  the  Task  Model  (CK),  including  task  hierarchies  for  all  agents  identified  above  (use 
IDEF3  to  represent  the  hierarchies); 

3.  Construct  the  Agent  Model  identifying  all  operator  and  system  agents  and  their  relationships; 

4.  Adapt  the  Task  Model  (CK)  to  a  five  level  Abstraction  Hierarchy  according  to  the  levels 
specified  by  the  EID  approach; 

5.  Generate  the  Task  Model  (EMD)  by  extending  the  Task  Model  (CK)  to  produce  task 
hierarchies  for  all  agents  using  PCT-based  hierarchical  goal  analysis; 

6.  Develop  the  User  Model  according  to  the  need  of  tracking  operator  preferences  and 
knowledge 

7.  Specify  the  content  of  the  System  Model  to  enable  representation  and  use  of  system 
preferences  and  knowledge; 

8.  Design  the  World  Model  to  contain  required  information  about  the  environment  necessary  for 
the  knowledge  system  to  operate  effectively; 

9.  Specify  the  Dialogue  Model,  Communication  Model  and  Co-ordination  Model  to  govern 
the  format  and  content  of  communication  among  agents  (ensure  that  the  ability  exists  for 
agents  to  provide  feedback  to  one  another); 

10.  Use  IDEF5  to  design  an  ontology  to  represent  the  contents  of  all  Explicit  Models; 

1 1 .  Develop  the  Knowledge  Model  to  encapsulate  the  ontology  and  an  associated  knowledge 
base  containing  information  from  all  Explicit  Models; 

12.  Within  the  Knowledge  Model,  represent  the  Task  Model  (EMD)  as  a  hierarchy  of  PCT 
loops  that  use  plan  recognition  and  plan  generation  to  form  input  perceptions  and  output 
behaviours 

13.  Create  the  Design  Model  to  produce  design  specifications  for  the  target  knowledge-based 

system;  and _ 
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14.  Apply  EID  techniques  to  develop  specifications  for  the  OMI  to  the  knowledge  system. 


Conclusions  for  IASs: 

1 .  Since  CommonKADS  was  developed  specifically  for  guiding  the  analysis,  design  and 
implementation  of  knowledge-based  systems,  it  is  very  well  suited  to  producing  such  a 
knowledge-based  system  for  UAV/UCAV  control. 

2.  IDEF3  is  recommended  for  temporal  and  process  modelling  in  the  UAV/UCAV  generic 
framework.  The  CommonKADS  approach  uses  UML  for  the  schematic  representation  of 
processes  and  associated  data,  agents,  tasks  and  inferences.  The  author  identifies  that  one 
of  the  drawbacks  of  UML  is  that  it  is  inflexible  in  representing  temporal  relationships  and 
constraints  among  elements.  Unlike  UML,  IDEF3  permits  flexible  modelling  of  temporal 
concepts.  The  integration  of  CommonKADS  and  IDEF  techniques  leads  to  a  robust 
framework  for  developing  knowledge-based  systems. 

3.  IDEF5  is  a  proven  useful  tool  to  design  an  ontology. 

4.  PCT  and  EMD  are  useful  for  determining  the  goals  an  operator  is  trying  to  achieve,  the  plans 
for  achieving  those  goals  and  how  a  system  can  assist  the  operator  most  effectively. 

5.  Stability  and  information  flow  analyses  attained  from  Hierarchical  Goal  Analysis  using 
principles  from  PCT,  could  contribute  to  the  generation  of  a  robust  goal  hierarchy  for  the 
UAV/UCAV  control  system. 

6.  Agents  offer  numerous  advantages  for  the  generic  framework.  Intelligent,  autonomous  agents 
are  ideally  suited  to  taking  over  some  tasks  for  human  operators,  serving  to  support  the  goals 
of  reduced  manning  and  enhanced  performance  in  a  complex  environment. 

7.  Ecological  Interface  Design  can  play  a  role  in  the  construction  of  an  adaptive  interface  for 
UAV/UCAV  control  by  developing  specifications  for  the  operator  interface  to  the  knowledge 
system. 


Hou,  M.  (2003).  Framework  for  Optimizing  Operator-Agent  Interaction.  A  preliminary  Technical 
Report  prepared  by  DRDC  Toronto. 


Overview: 

This  document  proposes  a  generic  framework  for  optimizing  operator-agent  interaction  based  on  a 
multiple-agents  architecture.  This  research  was  performed  in  the  context  of  controlling  multiple 
UAVs  in  a  complex  and  dynamic  environment. 

The  author  defines  an  Interaction  Model  as  a  need  to  reflect  the  work  environment  and  its  dynamic 
nature ,  as  perceived  by  the  operator  given  the  current  system  state  and  current  system  goals”. 

The  proposed  IAI  framework  is  composed  of  three  Senior  Agents: 

•  Managing  Agent:  This  agent  manages  the  information  flow,  and  the  control  of  the  display. 

The  agent  decides  the  automation  level,  and  coordinates  with  the  external  world,  modeling 
agent,  and  interaction  agents  based  on  the  states  (knowledge)  of  all  agents  (external  and 
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internal),  and  the  system  itself  (including  error  monitoring  and  emergency  control).  The 
management  agent  has  its  working  level  agent,  the  modeling  agent. 

•  External  Sensing  Agent:  This  agent  gathers  information  from  external  tasks,  sensors,  datalink 
and  keeps  the  managing  agent  and  modelling  agent  updated  on  the  current  “world  state”. 

•  Interaction  Agent:  This  agent  handles  the  information  from  the  interaction  with  operator  and 
communicates  with  the  managing  agent  to  control  the  display,  automation  levels  for  different 
agents,  3D  audio  and  speech  synthesis,  and  the  emergency  system  (with  feedback).  It 
includes  three  working  level  agents:  behaviour  agent,  perception  agent,  and  embedded 
cognition  agent. 

The  interaction  agent’s  working  level  agents  are: 

1 .  Behaviour  Agent  (uses  an  implicit  user  model):  This  agent  focuses  on  two  kinds  of  data 
inputs:  keyboard  and  mouse  or  other  input  devices.  The  data  would  provide  information  to 
the  cognition  agent  and  communicates  with  the  interaction  agent  about  the  cognitive  state 
of  the  operator  through  the  process  of  model  tracing. 

2.  Perception  Agent  (uses  an  implicit  user  model):  This  agent  focuses  on  low  arm 
movements,  facial  data  processing,  eye-gaze  tracking,  and  gestural  information.  The 
authors  advocate  that  the  operator’s  attention,  fatigue,  frustration,  and  even  fear  or 
excitement  can  be  interpreted  and  transferred  to  the  cognition  agent  for  further  analysis. 

3.  Cognition  Agent  (integrates  the  two  behavioural  user  models):  This  agent  focuses  on  the 
analysis  of  operator  workload,  situation  awareness,  complacency  and  skill  degradation 
(performance)  based  on  the  comparison  of  embedded  operator  models  with  the 
information  gathered  from  the  behaviour  agent  and  perception  agent. 
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Fizuz  6.  Adapt* tun  and  Aufimitinn  FraEM-work 


Conclusions  for  IASs: 

1 .  The  author  recommends  that  an  intelligent  adaptive  interface  should  incorporate  a  multi-agent 
system  with  the  following  models:  domain,  operator  task,  system,  dialogue  and  interaction. 

2.  The  interface  should  have  knowledge  of  the  operator,  system,  environment,  and  the  process 
of  various  tasks.  This  can  be  done  with  the  help  of  an  agent. 

3.  IAI  should  be  capable  of: 

a.  Adjusting  the  forms  of  information  transfer; 

b.  Transforming  the  information  contents; 
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c.  Altering/merging  modes  of  information  flow;  and, 

d.  Exchanging  /combing  communication  dialogue. 

4.  Function  allocation  should  be  applied  in  terms  of  answering  the  W5  (i.e.,  what,  who,  where, 
why,  when,  and  how)  question. 

a.  The  authors  proposed  that  that  function  allocation/adaptation  should  include  four  major 
processes:  knowledge  acquisition,  attention,  reasoning,  and  decision  making. 


3.5  Summary 

This  section  summarises  the  conceptual  frameworks  for  designing  intelligent  adaptive 
systems.  This  section  will  also  highlight  important  similarities  and  differences,  and 
advantages  and  disadvantages,  between  the  conceptual  approaches. 

3.5.1  Generic  Conceptual  Architecture  for  Intelligent  Adaptive  Systems 

Figure  6  describes  the  development  of  a  generic  conceptual  architecture  which  encompasses 
all  of  the  approaches  reviewed  in  Sections  8.2  through  8.4.  The  following  four  components 
are  common  to  all  IASs: 

•  Situation  Assessment  and  Support  System.  Comprises  functionality  relating  to  real¬ 
time  mission  analysis,  automation  and  decision  support.  The  system  monitors  and 
tracks  the  current  mission/goal  state,  aircraft/vehicle/system  status  (e.g.,  heading, 
altitude,  threats  etc.),  using  extensive  a  priori  task,  goal,  tactical,  operational,  and 
situational  knowledge.  Overall,  this  system  provides  information  about  the  objective 
(i.e.,  external)  state  of  the  aircraft/vehicle/system  within  the  context  of  a  specific 
mission,  and  uses  a  knowledge-based  system  to  provide  aiding  (e.g.,  automate  tasks) 
and  support  to  the  operator. 

•  Operator  State  Assessment.  Comprises  functionality  relating  to  real-time  analysis  of 
the  psychological,  physiological  and/or  behavioural  state  of  the  operator.  Primary 
functions  of  this  system  can  include  continuous  monitoring  of  workload,  inferences 
about  current  attentional  focus,  ongoing  cognition  (e.g.,  visual  and  verbal  processing 
load)  and  intentions,  using  extensive  a  priori  operator  knowledge  (e.g.,  models  of 
human  cognition,  control  abilities  and  communication). The  system  also  monitors  the 
operator  for  dangerously  high  and  low  levels  of  arousal.  Overall,  this  system  provides 
information  about  the  objective  and  subjective  (i.e.,  internal)  state  of  the  operator 
within  the  context  of  a  specific  mission.  This  information  is  used  to  optimise  operator 
performance  and  safety,  and  provides  a  basis  for  the  implementation  of  pilot  aiding 
and  support. 

•  Adaptation  Engine.  Utilises  the  higher-order  outputs  from  Operator  State  Assessment 
and  Situation  Assessment  systems,  as  well  as  other  relevant  aircraft/vehicle/system 
data  sources,  to  maximize  the  goodness  of  fit  between  aircraft/vehicle/system  state, 
operator  state,  and  the  tactical  assessments  provided  by  the  Situation  Assessment 
system.  These  integrative  functions  require  that  the  Adaptation  Engine  is  able  to 
influence  the  prioritisation  and  allocation  of  tasks  (i.e.,  intelligent  adaptive 
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automation)  and/or  determine  the  means  by  which  information  is  presented  to  the 
operator  (i.e.,  intelligent  adaptive  interface). 

•  Operator  Machine  Interface.  The  means  by  which  the  operator  interacts  with  the 
aircraft/vehicle/system  in  order  to  satisfy  mission  tasks  and  goals.  The  Operator 
Machine  Interface  is  also  the  means  in  which,  if  applicable,  the  operator  interacts  with 
the  IAS  (e.g.,  a  tasking  interface  manager).  The  design  of  the  OMI,  as  well  as  the 
automation,  is  defined  by  existing  HF  and  HCI  best-practice  and  standards. 

All  four  components  operate  within  the  context  of  a  closed-loop  system  insofar  as  there  is  a 
feedback  loop  that  re-samples  operator  state  and  situation  assessment  following  the  adaptation 
of  the  OMI  and/or  automation.  Similar  to  Perceptual  Control  Theory  (Powers,  1973;  Hendy  et 
al.,  2001),  the  goal  is  to  adjust  the  level  of  adaptation  so  that  optimal  operator  states  (e.g., 
performance,  workload  etc)  are  attained  and  maintained.  The  criteria  for  adaptation  (e.g., 
critical  events,  operator  state  and  behaviour)  are  described  in  the  next  section. 

3.5 .1 .1  Criteria  for  Implicit  A daptation 

In  Section  6.4.3. 1,  two  main  modes  of  control  over  function  allocation  were  described: 
explicit  allocation,  which  refers  to  situations  where  the  operator  has  the  control  over  the 
allocation  of  tasks;  and  implicit  allocation,  which  refers  to  the  machine  allocation  of  tasks 
automatically.  All  implicit  adaptive  systems  require  a  mechanism  by  which  changes  in  the 
OMI  and/or  levels  of  automation  are  implemented  or  triggered.  The  frameworks  reviewed  in 
this  section  utilise  implicit  adaptation  in  one  of  four  triggering  conditions: 

•  Critical  Events.  Critical  events  are  related  to  mission  goals;  if  critical  events  do  not 
occur,  automation  is  not  invoked.  Critical  event  logic  represents  the  least  technically - 
difficult  scheme  to  implement.  For  example,  if  a  specific  pre-de fined  critical-event 
occurs  (e.g.,  a  sudden  appearance  of  a  hostile  aircraft),  the  appropriate  defensive 
measures  are  performed  by  an  automated  system.  One  problem  is  that  it  is  often 
difficult,  particularly  in  a  complex  multi-task  environment,  to  adequately  specify  a 
priori  all  eventualities  that  may  occur  in  real  settings.  Another  related  problem  may 
arise  from  its  potential  insensitivity  to  the  current  needs  of  the  operator,  as  it  assumes 
a  priori  that  the  occurrence  of  a  critical  event  necessitates  automation  of  some 
functions  because  the  operator  cannot  efficiently  carry  out  these  functions  and  deal 
effectively  with  the  critical  event; 
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Knowledge  of  mission  plans/goals 
Knowledge  of  mission  time-lines 
Knowledge  of  mission  tasks/activities 
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Figure  6:  Generic  conceptual  architecture  for  Intelligent  Adaptive  Systems. 


Model  of  human  cognition 
Model  of  human  control  abilities 
Model  of  human  communication 
Model  of  human  interaction 
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•  Operator  State  and  Behaviour .  A  dynamic  assessment  of  operator  state  (e.g., 
workload)  and  behaviour  is  achieved  using  a  combination  of  psychophysiological  or 
behavioural  measures.  Changes  in  the  OMI  or  automation  are  predicated  upon  the 
momentary  assessment  of  operator  state  or  behaviour  where  the  violation  of  one  or 
more  pre-determined  criteria  triggers  the  adaptation.  The  main  advantage  of  this 
approach  is  that  the  measurement  is  real-time  and  reasonably  sensitive  to 
unpredictable  changes  in  operator  cognitive  states.  However,  this  approach  is  only  as 
accurate  as  the  sensitivity  and  diagnosticity  of  the  measurement  technology.  Further, 
due  to  the  latency  inherent  in  these  kinds  of ‘closed-loop  systems’  (i.e.,  feedback), 
state  and  behavioural  measurement  usually  occur  ‘after  the  fact’;  that  is,  it  follows  the 
point  in  time  when  an  adaptive  change  may  have  been  required  due  to  changes  in  task 
demands  or  operator  behaviour.  Finally,  in  highly  automated  systems  in  which  the 
operator  is  not  required  to  make  many  overt  responses,  behavioural  measurement  may 
be  so  impoverished  as  to  be  regarded  impractical; 

•  Operator  Modelling.  Operator  modelling  can  be  used  in  conjunction  with  operator 
state  and  behaviour  modelling,  whereby  the  adaptation  strategy  is  enhanced  by 
knowledge  of  human  cognition  and  action.  The  system  utilises  model(s)  of  human 
cognition  to  predict  (i.e.,  feed-forward)  the  human’s  performance  on  a  task,  and  to 
take  control  when  the  human  is  not  able  to  cope  with  task  demands.  Modelling 
techniques  have  the  advantage  that  they  can  be  implemented  off-line  and  be 
incorporated  into  on-line  operator  state  and  behaviour  adaptation  systems.  However, 
this  approach  is  only  as  good  as  the  theory  behind  the  model,  and  many  models  may 
be  required  to  deal  with  all  aspects  of  operator  performance  in  a  complex  task 
environment.  Many  approaches  to  adaptation  invoke  automation  or  OMI  adaptation 
on  the  basis  of  impending  performance  degradation,  as  predicted  by  a  human 
performance  model.  Models  can  be  classified  broadly  as  either  optimal  performance 
models  (i.e.,  signal  detection,  information  and  control  theories),  or  information 
processing  models  (i.e.,  such  as  multiple  resource  theory).  For  example,  an  adaptation 
strategy  based  on  the  multiple  resource  theory  (see  Wickens,  1984)  would  predict 
performance  degradation  whenever  concurrent  tasks  placed  excessive  demands  on 
finite  common  cognitive  resources.  When  the  combination  of  information  from  these 
different  sources  exceeds  some  threshold  in  the  algorithm,  tasks  are  allocated  to  the 
system  or  the  OMI  is  adapted  in  such  a  way  as  to  reduce  task  demands.  Adaptation 
based  on  cognitive  models  is  dependent  on  predictive  validity.  However,  no 
quantitative  model  of  multiple  resource  allocation  exists;  they  are  descriptive  models. 
Another  difficulty  associated  with  operator  modelling  is  that  it  requires  an  on-line 
assessment  of  what  the  operator  intends  to  do  (i.e.,  goal-tracking).  This  assessment  is 
especially  problematic  in  multi-task  situations;  the  activities  and  intentions  of 
operators  involved  in  a  single-task  would  be  obvious;  and, 

•  Hybrid.  These  systems  combine  performance  and  modelling,  or  critical-event  and 
performance,  or  other  possible  combinations  of  these  methods,  in  order  to  optimise 
their  relative  benefits  (Hilburn  et  al.,  1993).  Hilburn  et  al.  propose  that  a  hybrid 
system  incorporating  more  than  one  of  these  methods  might  optimise  their  relative 
effectiveness  (i.e.,  optimise  the  benefits  and  minimise  the  drawbacks).  Figure  7 
describes  a  hybrid  implicit  adaptation  system  in  which  the  system  compares  operator 
behaviour  with  beliefs  about  the  operator’s  intent,  which  then  drives  decision  aiding 
and  interface  adaptation.  The  system  model  of  the  operator’s  interaction  with  the 
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system  is  based  on  a  combination  of  knowledge  of  critical  events  and  models  of 
human  cognition.  A  probability  distribution  about  the  operator’s  goals  and  intent  (bar 
graph)  is  computed  from  this  knowledge  and  observations  of  the  operator.  Actions, 
such  as  task  automation  or  interface  adaptation,  are  then  generated  based  on  their 
expected  utility. 


*  User  activity 

*  User  profile 

*  User  utterances  ► 

*  Data  stuctures 

*  Context 


Utility-directed  actions 


Figure  7:  Hybrid  implicit  adaptation  system  (Horwitz,  1999,  p.  18). 

3.5.2  Comparison  of  Conceptual  Frameworks 

Table  6,  Table  7,  and  Table  8  summarise  the  advantages  and  disadvantages  of  the 
frameworks,  the  relationship  between  the  frameworks  in  terms  of  authority,  agency  and  user 
model,  and  which  frameworks  are  applicable  to  a  given  situation  (i.e.,  domain  applicability). 

Table  9  provides  a  summary  of  IAH,  IAI,  and  IAA  frameworks. 
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Table  6:  Summary  of  Intelligent  Adaptive  Interface  Frameworks. 


Adaptive  Icon  Toolbar 

ConCall 

SAWA  (USA) 

Work-Centered  Decision 
Support 

Advantages 

Adaptation  of  the  OMI  can  increase 
efficiency  of  customization  and 
learning  of  interface  features. 

Features,  functions,  and  content  can 
be  provided  at  the  right  time. 

Useless  features  can  be  removed. 

Compensate  for  individual 
characteristics. 

Provide  context  dependent  help. 

Content  can  be  provided  at  the  right 
time. 

Compensate  for  individual 
characteristics. 

An  offline  system  for  decision 
making;  therefore  no  real-time, 
criticality  issues. 

Semantic  Web  technologies  can  be 
used  for  representing  and  reasoning 
about  knowledge  pertinent  to  a 
situation’s  domain. 

Content  can  be  provided  at  the  right 
time. 

Compensate  for 

individual/population  characteristics. 

Provide  context  dependent 
information. 

Decrease  mental  workload. 

A  work  domain  ontology  was  useful  as 
the  organizing  framework  across  the 
decision  support  tool  set. 

Minimizes  perceptual,  cognitive,  and 
motor  task  demands  associated  with 
identifying,  seeking,  or  interpreting 
relevant  information  and  producing 
work  artefacts. 

Automated  agents  need  to  be 
observable  (or  transparent/visible)  so 
that  operators  are  able  to  see  what  the 
automated  agents  are  doing  and 
understand  what  they  will  do  next 
relative  to  the  state  of  the  task. 

Ecological  Interface  Design  is  an 
effective  means  of  providing  work- 
centered  decision  support. 

Cognitive  work  analysis  (CWA)  as  part 
of  the  human-centered  design  process 
is  an  effective  means  of  establishing 
system  requirements  to  ensure  a 
human-centered  system. 

Provide  context  dependent  information 

Compensate  for  individual 
characteristics. 

Disadvantages 

Potential  for  poor  user  models. 

Lack  of  transparency  and 
predictability. 

Lack  of  controllability  of  the  system. 

Potential  for  poor  user  models.  Too 
many  recommendations  therefore 
operators  stopped  using  it  out  of 
frustration.  Due  to  a  lack  of  a  proper 
mental  model  of  the  system, 
operators’  were  found  to  request  a 
function  that  would  undo  a  previous 
adaptive  change  that  the  system  has 
determined  would  be  right  for  the 

Semantic  Web  Technologies  are 
difficult  to  implement. 

How  much  “visibility”  needed  is 
questionable  (i.e.,  not  at  all  but  then 
issue  of  trust  and  mistrust  can  occur  or 
fully  visible  such  as  the  Microsoft 
“PaperClip”  which  takes  advantage  of 
assistant  and  subordinate  metaphors). 

Potential  for  poor  task  models. 
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Adaptive  Icon  Toolbar 

ConCall 

SAWA  (USA) 

Work-Centered  Decision 
Support 

operator. 

Lack  of  controllability  of  the  system. 
Operators  also  wanted  an  UNDO 
function  and  wanted  to  be  less 
disturbed. 

Authority 

The  operator  has  authority  over 
analysis  and  implementation  of 
adaptation  while  the  system  has 
authority  over  acquiring  the 
information.  A  shared  role  for 
implementing  adaptation. 

The  operator  has  authority  over 
analysis  and  implementation  of 
adaptation  while  the  system  has 
authority  over  acquiring  the 
information.  A  shared  role  for 
implementing  adaptation. 

Operator  always  in  control  of 
responding  while  the  system 
automatically  analyzes  and 
synthesizes  information  and 
provides  recommendations. 

Operator  control  over  implementation  of 
recommendations. 

Agent 

The  system  performs  all  roles. 

The  system  performs  all  roles. 

System  analyses  and  synthesizes 
information  and  makes  decision 
recommendations  but  operator 
implements  decision  making. 

System  acquires  and  analyzes 
information  and  makes 
recommendations. 

User  Model 

Operator  behaviour. 

Explicit  user  model. 

Monitors  evolving  situation  but  not 
operator  characteristics. 

Based  on  task  model. 

Domain 

Applicability 

Web  and  stand-alone  applications, 
features,  functions,  and  content  can 
be  provided  at  the  right  time. 

Web  and  stand-alone  applications, 
features,  functions,  and  content  can 
be  provided  at  the  right  time. 

Aviation. 

Decision  aiding/information 
management. 

Decision  making,  information 
management. 

Stock  Trader 

Personal  Web  Searcher 

DIAManD 

Advantages 

An  implicit  user  model  is  an  effective 
and  non-obstructive  means  of 
constructing  a  user  model. 

Adaptation  of  the  OMI  can  increase 
efficiency  of  customization  and 
learning  of  interface  features  and  of 
stock  trading.  That  is,  as  the  operator 
began  to  better  understand  how  the 
system  works,  they  began  to  accept 
more  recommendations. 

Mixed-initiative  interfaces  (e.g., 
direct  interaction  with  an  agent)  can 
increase  situational  awareness  and 
develop  better  mental  model  of  the 
system. 

Content  can  be  provided  at  the  right 
time. 

Useless  features  can  be  removed. 

Compensate  for  individual 

A  mixed-initiative  framework  (e.g., 
DIAManD)  in  which  the  learner 
and  human  operator  are  each 
participants  in  a  dialogue  could 
improve  the  learner's  hypothesis 
with  minimal  effort  on  the  part  of 
the  operator. 

Adaptation  of  the  OMI  can 
increase  efficiency  of 
customization  and  learning  of 
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Stock  Trader 

Personal  Web  Searcher 

DIAManD 

Compensate  for  individual 
characteristics. 

Provide  context  dependent 
information. 

characteristics. 

Provide  context  dependent 
information. 

interface  features. 

Compensate  for  individual 
characteristics. 

Provide  context  dependent 
information. 

Disadvantages 

Burden  the  operator  with  an  increased 
workload  by  having  to  either  decide  or 
implement  the  adaptation. 

Potential  for  human  OOTL 
performance  problems. 

Potential  for  poor  user  models. 

Potential  for  human  OOTL 
performance  problems. 

Burden  the  operator  with  an 
increased  workload  by  having  to 
either  decide  or  implement  the 
adaptation. 

Obtrusive. 

Difficult  to  maintain  the  operator 
“in-the-loop”. 

Authority 

Operator  control  over  implementation 
of  recommendations. 

Operator  control. 

Operator. 

Agent 

System  acquires  and  analyzes 
information  and  makes 
recommendations. 

System  analyzes  and  makes 
recommendations. 

System. 

User  Model 

Based  on  operator  behaviour. 

Based  on  operator  behaviour. 

Monitors  evolving  situation. 

Based  on  operator  behaviour  and 
task  model. 

Domain 

Search  engine,  information 

Search  engine,  information 

Decision  making,  information 

Applicability 

management,  decision  aiding. 

management,  decision  aiding. 

management. 
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Table  7:  Summary  of  Intelligent  Adaptive  Automation  Frameworks. 


CAMA  (Germany) 

Co-Pilote  Electronique  (France) 

Playbook 

Advantages 

Mainly  for  enhancing  situational  awareness. 

Content  can  be  provided  at  the  right  time. 

Compensate  for  individual  characteristics. 

Provide  context  dependent  help. 

Decrease  mental  and  physical  workload. 

Tasked  systems  are  always  sub-ordinate,  but  know 
enough  about  the  tasks  in  the  domain  that  instructing 
them  is  vastly  easier  than  instructing  traditional 
automated  systems. 

Enabling  a  system  to  behave  more  like  an  intelligent 
subordinate,  operators  may  be  more  tolerant  of  their 
weaknesses  and  acceptable  of  their  capabilities  in  a 
controlled  setting  (operator  acceptance). 

Playbook  provides  a  complete  architecture  for  the 
integration  of  human  input,  intelligent  a  priori  planning, 
reactive  planning  and  event  handling,  and  ongoing 
automation  control  loops. 

Affords  collaboration  between  interfaces  and  operators 
in  order  to  achieve  the  operator’s  goals. 

Disadvantages 

Still  has  many  modules  (16)  and  therefore 
complex  to  implement. 

Potential  for  human  OOTL  performance 
problems. 

Potential  for  human  OOTL  performance 
problems. 

Tasking  interfaces  should  not  rely  on  a  predefined  set 
of  task  models.  The  operator  should  be  able  to  create 
novel  tasks  and  to  store  components  of  models  which 
are  useful. 

Operators  need  sufficient  training  for  interacting  with 
the  tasking  interface. 

Authority 

Sharing  of  initiation  but  human  has  ultimate 
control. 

Operator  always  in  control  of  implementation. 

Operator  always  in  control. 

Agent 

Mostly  the  system  except  for  responding. 

System  performs  the  analysis  and  decision 
recommendations  while  the  operator  implements 
decisions. 

Sharing  of  roles  between  system  and  human. 

User  model 

Functions  for  situation  assessment  and  mission 
planning  make  up  the  core  of  tactical  mission 
management  systems. 

Based  on  a  shared  task  model. 

Domain 

Applicability 

Aviation. 

Aviation. 

Decision  aiding/information  management. 
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CAMA  (Germany) 

Co-Pilote  Electronique  (France) 

Playbook 

(i.e.  what, 
when  &  how) 

Maintain  situational  awareness. 

Decision  aiding/information  management. 

Decision  aiding/information  management. 

Delegation  of  tasks. 

Intelligent  Classroom 

LookOut 

Advantages 

Human-machine  cooperation  can  be  achieved  by 
allowing  an  operator,  in  executing  her  part  of  a 
plan,  to  expect  a  system  to  help  in  executing  part 
of  the  plan.  A  plan  is  a  set  of  processes  (often  to 
be  executed  by  a  number  of  different  agents) 
that  when  run  together  successfully,  accomplish 
some  goal.  Plans  attempt  to  express  a  common 
understanding  of  how  a  speaker  and  an  audio/ 
visual  assistant  interact. 

The  more  the  system  understands  its  operators 
and  their  tasks,  the  more  useful  it  will  be  for 
them.  This  requires  a  learning  module. 

The  same  techniques  implemented  in  the 

Intelligent  Classroom  can  be  applied  to  a  broad 
range  of  interactive  applications. 

Adaptive  automation  performed  in  real-time. 

Adaptive  automation  performed  in  real-time. 

The  system  is  designed  to  continue  to  learn  from 
operators  through  caching  operator  behaviour 
with  the  system  and  by  the  operator  specifying  a 
policy  for  continual  learning  (e.g.,  set  system  to 
cache  behaviour  at  particular  times. 

The  decision  of  automation  initiation  is  based  on 
when  an  agent  believes  that  they  will  have 
greater  expected  value  than  inaction  for  the 
operator,  taking  into  consideration  the  costs, 
benefits  and  uncertainties  in  the  operator’s 
goals. 

Disadvantages 

Potential  for  poor  user  models. 

Obtrusive. 

May  be  difficult  to  maintain  the  operator  “in-the- 
loop,  and  may  lead  to  OOTL  performance 
problems. 

Potential  to  increase  workload  (operator  may 
have  to  decide  to  implement  and/or  automation 
implementation  is  inappropriately  applied). 

Operator  may  feel  out  of  control. 

Potential  for  poor  user  models. 

Potential  to  increase  workload  (operator  may 
have  to  decide  to  implement  and/or  automation 
implementation  is  inappropriately  applied). 

May  be  difficult  to  maintain  the  operator  “in-the- 
loop,  and  may  lead  to  OOTL  performance 
problems. 

Obtrusive 

Operator  may  feel  out  of  control. 

Authority 

System/Human  share  tasks  dynamically. 

The  system  performs  all  tasks  but  the  operator 
has  control  over  acquiring  operator  specified 
information  and  shares  the  role  of  taking  action 
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Intelligent  Classroom 

LookOut 

to  system  recommendations. 

Agent 

System. 

System. 

User  model 

Monitors  evolving  situation. 

Based  on  operator  behaviour  and  task  model. 

Inferred  probability  of  a  user  goal  (hierarchical 
goal  model). 

Domain 
Applicability 
(i.e.  what, 
when  &  how) 

Assistant  aiding  for  execution  of  tasks  (non- 
delegated). 

Increase  efficiency  of  task  (e.g.,  time  saving). 

Assistant  aiding  for  execution  of  tasks  (non- 
delegated). 

Increase  efficiency  of  task  (e.g.,  time  saving). 
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Table  8:  Summary  of  Intelligent  Adaptive  Hybrid  Frameworks. 


Cognitive  Cockpit  (UK) 

PA/RPA  (USA) 

CASSY  (German) 

DRDC  IAI  Framework 
(Canada) 

Advantages 

Feedback/Feed-forward  architecture 
allows  system  adaptiveness 
(dynamic  function  allocation). 

Functional  analysis  of  cognitive 
work  provides  essential  foundations 
for  the  successful  development  and 
implementation  of  cognitive  cockpit 
technologies  for  pilot  aiding. 

Showed  that  interfaces,  tasks,  and 
automation  can  be  managed  by  a 
tasking  interface  system  based  on  a 
shared  task  model. 

Affords  collaboration  between 
systems  and  operators  in  order  to 
achieve  the  operator’s  goals. 

More  emphasis  on  development  of  OMI. 

Benefits  of  an  explicit,  integrative  framework 
(task  model).  Having  a  task-based  framework 
proved  to  be  an  effective  means  of 
coordinating  and  minimizing  the  costs  of 
revising  or  extending  multiple,  diverse  sets  of 
subsystems. 

The  use  of  Intelligent  Object-Oriented  Design 
(IOOD)  provides  a  series  of  steps  to  transform 
requirements  to  more  abstract  views  of 
objects. 

Iterative  development  is  most  productive  when 
supported  by  a  set  of  management,  design 
and  testing  tools  (e.g.,  Plan  Goal  Graph  Tool 
(PGG);  Display  Analyst). 

Affords  collaboration  between  systems  and 
operators  in  order  to  achieve  the  operator’s 
goals. 

Less  complex  framework. 

Less  resources  needed  (e.g.,  no 
psycho-physiological  measures). 

Reduced  human  OOTL 
performance  problems. 

Algorithms  based  on  a-priori 
probabilities  for  possible 
hypotheses  have  proven  useful  for 
recognizing  and  estimating 
operator  intent. 

UAV  Simulation  Tests  found  that : 

•  Operators  at  all  crew  positions 
performed  more  effectively  from 
both  quantitative  and  qualitative 
perspectives  when  the  IAI  multi¬ 
agent  system  was  selected  ON. 

•  When  IAI  was  ON,  CTSs  were 
shortened,  less  tasks  were  shed, 
the  UAV  trajectory  scores  were 
better  and  much  less  time  was 
spent  in  the  no-fly  areas. 

•  Operators’  overall  SA  was 
improved  and  overall  workload 
was  reduced  as  well. 

Affords  collaboration  between 
systems  and  operators  in  order  to 
achieve  the  operator’s  goals. 

CommonKADS,  IDEF  standards, 

EMD,  PCT  and  PACT  are  useful  for 
developing  agents. 

Agents  offer  numerous  advantages 
(e.g.,  automation). 

Disadvantages 

Highly  complex  and  difficult  to 
implement. 

Analysis  is  in-depth,  time- 
consuming  and  requires  trained  HF 
professionals. 

Limited  resources  due  to  complexity 
(although  this  is  changing  with 
time). 

Lack  of  transparency  and 

Highly  complex  and  difficult  to  implement. 

Analysis  is  in-depth,  time-consuming  and 
requires  trained  HF  professionals. 

Limited  resources  due  to  complexity  (although 
this  is  changing  with  time). 

Lack  of  transparency  and  predictability. 

Still  very  complex  that  requires  a 
lot  of  analysis. 

UAV  simulation  tests  found  that: 

•  The  crews  came  in  for  only 
two  days  and  the  novelty  of 
the  conventional  interface 
would  not  have  worn  off. 

•  The  crewmembers  would 
rather  work  with  the  interface 
manually  than  give  up  control 
to  the  IAI. 

•  The  conventional  interface 
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predictability. 

was  designed  to  be  effective, 
however  the  IAI  ON  interface 
still  performed  better. 

•  Participants  had  been  trained 
on  the  conventional  interface 
for  which  they  had  developed 
work  strategies  and  not  on  the 
IAI.  With  the  IAI  functionality 
selected  ON,  the  participants 
may  rely  on  the  original  work 
strategies  because  they  were 
known  and  effective. 

Highly  complex  and  difficult  to 
implement. 

Analysis  is  in-depth,  time-consuming 
and  requires  trained  HF 
professionals. 

Authority 

More  emphasis  on  automation  of 
tasks  based  on  feedback  and  feed¬ 
forward  architecture. 

Operator  always  in  control  except  in 
critical  conditions  (e.g.,  autopilot 
turns  on  when  pilot  looses 
consciousness). 

More  system  autonomy  for  analysis  and 
recommendations  for  decisions  but  operator 
always  in  control,  especially  for 
implementation  of  automation. 

Operator  always  in  control  of 
implementation. 

Human  always  in  control  except  when 
analyzing  and  synthesizing  the 
information. 

Goal-driven  control  loop  for  the 
initiation  of  automation. 

Agent 

Full  sharing  of  tasks  between 
system  and  operator. 

More  emphasis  on  supporting  the  pilot  with  his 
task  (shared  knowledge  to  plan  and  suggest 
courses  of  action  and  to  adapt  cockpit 
information  displays  and  the  behavior  of 
aircraft  automation). 

System  performs  most  of  the 
tasks  under  the  control  of  the 
human  except  for  responding. 

Sharing  of  tasks  between  human  and 
system  except  for  analysis. 

User  model 

More  emphasis  on  operator  state 
(workload). 

Tracks  goals/tasks  implicitly  and 

Less  emphasis  on  operator  state. 

CIM  Intent  estimation  is  used  to  assess  if 
system  action  is  required  (to  assess  pilot 

Operator  behaviour  (crew  actions 
are  derived  indirectly  by 
interpreting  aircraft  data  for  pilot 

Task/function  model  but  no 
monitoring  of  operator  behaviour  or 
state. 
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updates  as  necessary. 

Intent  estimation  is  used  to  assess  if 
system  action  is  required. 

intent). 

intent  and  error  recognition. 

Interaction  agents  (behaviour, 
perception  and  cognition)  (Hou, 

2003). 

Plan  recognition  and  generation  are 
used  in  conjunction  with  pre-defined 
goals/tasks  and  matched  to  a  World 
state  to  assess  if  system  action  is 
required. 

Similarities 

Similar  frameworks  based  on  several  modules:  task/function-based,  environmental,  mission  and  operator  characteristics  by  monitoring,  assessing  and  implementing 
adaptive  interface  and/or  automation. 

Potential  for  poor  user  models  (Does  not  matter  what  the  model  is  based  on.  I.e.,  psycho-physiological  measures  or  plan  generation  and  recognition/intent 
estimation). 

Lack  of  controllability  of  the  system. 

Burden  the  operator  with  an  increased  workload  by  having  to  either  decide  or  implement  the  adaptation. 

Obtrusive. 

Domain 

Applicability 

Aviation,  UAV. 

Increase  task  efficiency. 

Decision  aiding/information  management. 

Managing  increasing  complex  systems. 

Sharing  of  tasks/functions. 

98 


Table  9:  Summary  of  Intelligent  Adaptive  Interface,  Automation  and  Hybrid  frameworks. 


Intelligent 

Adaptive 

Hybrid 

•  Domain  Areas:  Aviation,  UAV,  decision  aiding/information  management. 

•  Adaptive  automation  (of  tasks)  more  prevalent. 

•  Comprehensive. 

•  Critical  (more  serious  consequences  of  errors). 

•  Real-time  adaptation. 

•  Operators  are  more  likely  to  have  authority  over  implementation  of  automation,  or  adaptation  of  OMI. 

•  User  model:  usually  very  complex  (includes  many  factors);  usually  elicited  implicitly  (physiologically-based  on 
workload  principles). 

Intelligent  Adaptive 
Interface 

•  Domain  areas:  adaptive  and  adaptable  Ul,  applications  (e.g.,  web  and  standalone),  search  engines,  information 
management. 

•  Involves  a  more  narrow  scope  (only  perform  some  of  the  roles,  not  always  all  of  them) 

•  Non-critical  applications. 

•  Learning  IAI  (algorithms)  is  more  prevalent  (suspect  it  is  easier  to  implement  because  systems  are  less 
complex,  and  not  real-time). 

•  User  model:  less  complex  if  elicited  implicitly  then  more  often  behaviour-based. 

Intelligent  Adaptive 
Automation 

•  Adaptive  automation  performed  in  real-time. 

•  Non-critical  applications. 

•  Operators  have  authority  over  implementation  of  automation. 

•  Provide  context  dependent  help. 
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4  Analytical  Techniques 


4.1  Introduction 

Analytical  approaches  such  as  Mission  Function  Task  Analysis  (MFTA),  Cognitive  Work 
Analysis  (CWA)  and  Applied  Cognitive  Work  Analysis  (ACWA)  can  provide  the  OMI 
communication,  visual  display  and  control  requirements  needed  for  the  design  of  intelligent 
adaptive  systems  as  well  as  a  functional  decomposition  of  the  tasks  within  the  domain 
envisaged  for  it.  In  addition,  they  provide  the  means  to  capture  more  detailed  knowledge  from 
Subject  Matter  Experts  for  embedding  in  an  intelligent  knowledge  based  system.  In  order  to 
intelligently  adapt  the  interface  to  an  operator’s  needs,  the  system  must  be  privy  to  the  current 
mission  status,  platform  status  (e.g.,  heading,  altitude  and  threat)  and  also  be  invested  with 
extensive  a  priori  tactical,  operational  and  situational  knowledge.  This  provides  information 
about  the  objective  state  of  the  platform  within  a  mission/goal  context  and  provides  a  basis  for 
the  adaptation  of  the  interface  to  support  the  operator. 

To  achieve  its  goals,  the  design  of  an  IAS  requires  a  validated  requirements  gathering  and 
analysis  methodology  to  provide  a  comprehensive  view  and  understanding  of  the  tasks  and 
cognitive  processes  involved  in  the  complex  environment  that  will  include  an  intelligent 
adaptive  systems.  The  following  review  will  provide  a  template  and  the  basis  for  an 
understanding  of  other  published  methods  in  order  to  recommend  the  methodology  or 
methodologies  that  will  best  support  the  capture  and  analysis  of  processes,  tasks  and 
requirements  for  future  IAS  design  activities. 

4.2  Analysis  Methodology 

The  section  reviews  the  following  analysis  methodologies: 

•  Hierarchical  Task  Analysis  (HTA); 

•  Mission,  Function  and  Task  Analysis; 

•  Hierarchical  Goal  Analysis  based  on  the  Perceptual  Control  Theory; 

•  Goal-Directed  Task  Analysis  (GDTA); 

•  Cognitive  Task  Analysis  (CTA); 

•  Team  Cognitive  Task  Analysis  (Team  CTA); 

•  Applied  Cognitive  Task  Analysis  (ACTA); 

•  Cognitive  Work  Analysis;  and, 

•  Applied  Cognitive  Task  Analysis. 
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4.2.1  Hierarchical  Task  Analysis 


Hierarchical  Task  Analysis 


Reference: 

Crystal,  A.  &  Ellington,  B.  (2004).  Task  Analysis  and  Human  Computer  Interaction:  approaches, 
techniques,  and  levels  of  analysis.  Proceedings  of  the  Tenth  Americas  Conference  on  Information 
Systems,  New  York,  New  York. 

Hone,  G.  &  Stanton,  N.  (n.d.).  HTA:  The  Development  and  Use  of  Tools  for  Hierarchical  Task 
Analysis  in  the  Armed  Forces  and  Elsewhere.  Accessed  at 
http://www.hfidtc.com/pdf/reports/HTA%20report.pdf,  January  28,  2007. 

Miller,  C.  &  Vicente,  K.  (2001).  Comparison  of  Display  Requirements  Generated  via  Hierarchical 
Task  and  Abstraction-Decomposition  Space  Analysis  Techniques.  International  Journal  of 
Cognitive  Ergonomics  5(3),  335-355. 

Stanton,  N.  (n.d.).  Hierarchical  Task  Analysis:  Developments,  Applications,  and  Extensions. 
Accessed  at  http://www.hfidtc.com/pdf/reports/HTA%20Literature%20Review.pdf.  January  28, 
2007. 


Overview: 

Hierarchical  Task  Analysis  is  a  broad,  simple  and  informal,  and  representationally  streamlined  task 
analysis  method.  HTA  describes  a  system  in  terms  of  its  goals,  which  are  expressed  in  terms  of 
some  objective  criteria.  HTA  is  able  to  describe  a  system  in  terms  of  the  tasks  conducted  by 
individuals,  as  well  as  producing  a  systems  analysis.  Thus,  HTA  is  able  to  describe  human  and 
non-human  tasks  performed  by  a  system. 

HTA  focuses  on  what  an  Operator  is  required  to  do,  in  terms  of  actions  and/or  cognitive  processes 
to  achieve  a  system  goal.  Task  knowledge  is  structured  in  a  hierarchical  action  means-ends 
relationship  (how  subtasks  may  be  composed  to  accomplish  higher  level  tasks),  and  sequential 
relationships  (how  tasks  must  be  performed  temporally)  (Miller  &  Vicente,  2001). 

Process: 

The  method  describes  a  task  in  terms  of  a  hierarchy  of  operations  and  plans,  and  identifies  the 
conditions  under  which  the  sub-tasks  should  be  completed  in  order  to  meet  the  system  goals.  The 
methods  result  in  a  hierarchy  of  three  levels  of  task  analysis,  including: 

•  Goals.  A  goal  a  human  wants  to  achieve; 

•  Tasks.  Structured  set  of  tasks  and  plans  that  are  outlined  in  a  sequence  to  achieve  the 
goal;  and, 

•  Operations  or  Actions.  Simple  tasks  that  have  no  further  structure/plan;  these  are  the 
lowest  level  of  decomposition. 

HTA  views  tasks  in  a  more  abstract  sense,  as  a  set  of  interlinked  goals,  resources,  and  constraints 
(Crystal  &  Ellington,  2004). 


3  Note  that  for  this  reference,  and  all  subsequent  references,  these  icons  relate  only  to  the  publications 
cited. 
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Miller  &  Vicente  (2001)  indicate  that  HTAs  are  generally  presented  in  two  formats: 

•  Graphical  format  shows  the  hierarchical  and  aggregate  relationship  between  tasks.  Each  layer 
of  the  hierarchy  represents  a  series  of  tasks/actions  which  accomplish  the  higher  level  task.  A 
plan  is  always  placed  along  the  vertical  line  connecting  the  lower  level  tasks  to  the  higher  level 
tasks  to  identify  how,  when,  and  in  what  order  they  must  be  performed  to  accomplish  the 
higher  level  task.  The  plan  outlines  the  parallel  or  sequential  relationships  amongst  the  tasks; 
and, 

•  Tabular  format  with  progressive  indenting  and  task  numbering  used  to  track  task 
decomposition.  This  format  may  make  it  more  difficult  to  visualize  task  relationships;  however, 
it  facilitates  additional  information  links  to  other  tasks  (e.g.,  duration  or  frequency  information, 
potential  for  human  errors,  resources  required,  etc.). 

HTA  information  sources  include:  operator  interviews,  direct  observation,  and  training  or 

procedural/operational  manuals. 


Advantages: 

1 .  Cost  efficient  task  analysis  method; 

2.  Easy  to  learn  and  apply; 

3.  Results  can  provide  the  input  to  a  variety  of  other  analyses;  and 

4.  Provides  an  analytical  framework  for  designers. 

Disadvantages: 

1 .  Provides  a  narrow  view  of  the  task,  and  should  normally  be  used  in  conjunction  with  other  task 
analysis  methods  to  increase  its  effectiveness,  and  to  develop  a  more  complete  understanding 
of  human  activity. 

2.  Generally  used  to  describe  simple  rather  than  complex  tasks. 

IAI  Applicability: 

1 .  Provides  a  model  for  task  execution,  enabling  interface  designers  to  envision  the  goals,  tasks, 
subtasks,  operations,  and  plans  essential  to  operators’  activities  (Crystal  &  Ellington,  2004). 

2.  Can  be  easily  extended  to  provide  system  and  information  requirements. 

3.  Information  needs  (both  input  and  output)  are  typically  deduced  for  the  tasks.  These  needs, 
when  combined  with  task  relationship  information,  can  provide  a  basis  for  prioritizing, 
clustering,  filtering,  or  sequencing  information  presentation  in  an  interface  design  (Miller  & 
Vicente,  2001). 
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4.2.2  Mission,  Function,  and  Task  Analysis 


Mission,  Function,  and  Task  Analysis 


Reference: 

Chow,  R.,  Kobierski,  B.,  Coates,  C.  &  Crebolder,  J.  (2006).  Applied  Comparison  between 
Hierarchical  Goal  Analysis  and  Mission,  Function,  and  Task  Analysis.  Proceedings  of  the  Human 
Factors  and  Ergonomics  Society  50th  Annual  Meeting. 

Darvill,  D.,  Kumagai,  J.  &  Youngson,  G.  (2006).  Requirements  Analysis  Methods  for  Complex 
Systems.  Technical  Report  to  Defence  Research  &  Development  Canada  Valcartier  (CR  2005-076). 


Overview: 

MFTA  is  a  top-down  analysis  that  is  generally  used  during  the  initial  stages  of  systems  development 
to  specify  requirements.  This  concept  promotes  a  link  between  analysis  and  design,  and  validation 
and  verification  are  completed  at  each  step  of  the  analysis. 

Baseline  scenarios  are  analysed  to  produce  a  composite  scenario  (mission)  that  identifies  all 
relevant  and  important  functions  of  the  system  (Chow  et.  al.,  2006).  A  mission  is  decomposed  into 
mission  segments,  which  are  then  further  decomposed  into  functions,  followed  by  lower-level 
functions.  Decomposition  may  be  completed  according  to  functional  groupings  (e.g.,  navigation, 
communication),  or  different  points  along  a  timeline  (e.g.,  take-off,  cruise,  landing). 

At  the  lowest  level,  function  allocation  is  completed  to  assign  a  function  to  either  a  human  or  a 
system.  Additional  information  can  also  be  linked  to  each  task  and  function,  such  as: 

•  Completion  time; 

•  Relevant  perceptual  and  cognitive  tasks; 

•  Skills  and  knowledge  required;  and, 

•  Identification  of  critical  tasks. 

Process: 

MFTA  is  generally  structured  according  to  the  following: 

•  Mission  analysis  and  scenario  development,  defines  the  overall  requirements  of  the  system.  The 
system  is  described  in  terms  of  its  operational  requirement  (what  the  system  must  do),  and  the 
environment  or  scenario  under  which  the  operational  requirements  must  be  completed.  The 
mission  is  described  in  detail,  to  identify  important  mission  phases,  system  functions,  activity 
timeline,  and  external  events  which  may  impact  the  system; 

•  Function  analysis,  the  system  is  analysed  in  terms  of  the  functions  which  must  be  performed. 
Function  analysis  is  structured  according  to  a  top-down  hierarchy.  Decomposition  of  functions  is 
graphically  represented  via  Function  Flow  Diagrams,  outlining  the  operational  characteristics  of 
the  mission.  Function  decomposition  is  completed  when  the  task  level  is  attained; 

•  Function  allocation.  Functions  are  allocated  to  a  system  component  or  to  a  human.  Functional 

allocation  identifies  whether  the  system  can  adequately  support  the  execution  of  the  mission, 
and  corresponding  functions.  If  deficiencies  are  identified  in  the  baseline  system,  alternate _ 
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allocations  are  proposed  and  assessed.  Function  allocation  analyses  provide  the  basis  for 
subsequent  efforts  relating  to  task  analysis  and  description,  performance  analysis,  display  and 
control  selection,  or  crew-station  design  (Darvill  et.  al.,  2006). 

•  Task  analysis:  Defines  what  an  Operator  is  required  to  do,  and  identifies  the  interaction  between 
the  Operator  and  the  system.  Task  analysis  permits  the  application  of  relevant  knowledge  on 
human  performance  (Darvill,  2006).  A  task  analysis  may  characterize  each  task  according  to  the 
following  elements: 

•  Task  description; 

•  Task  completion  time; 

•  Action  requirement; 

•  Information  requirement;  initiating  conditions; 

•  Feedback; 

•  Cognitive  processing  requirements; 

•  Decision  requirements; 

•  Priority; 

•  Criticality; 

•  Knowledge; 

•  Skills;  and 

•  Ability. 

Performance  prediction,  predicts  how  well  an  Operator  will  perform  a  task.  The  mission,  function, 
and  task  analysis  results  are  linked  to  system  performance  criteria.  These  criteria  confirm  the 
function  allocation  to  Operators  or  the  system. 

Darvill  et.  al.,  (2006)  provides  a  graphical  ‘waterfall’  relationship  of  MFTA  elements,  as  illustrated 
below. 
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Mission  and 
Scenario  Analysis 


■objectives 
■roles  and 
responsibilities 
-factors  affecting 
the  mission 
■co-operating 
agencies 
■ROE 
■Threat 

■Environmental 

conditions 


Function  Analysis 


■Functions 
corresponding  to 
mission  phases 
■Decomposed  to 
lower  level 
functions 


Function 

Allocation 


■Operator/ 
maintainor 
functions 
corresponding  to 
mission  phases 


Task  Analysis 


■Task  and  critical 
task  information 


Performance 

Prediction 


■Task  sequences 

iQSJi?) 

■Predicted 

performance 

■Error  analysis 

■Mission 

completion 

analysis 

■Design 

requirements 


Advantages: 

1 .  Relatively  easy  method  for  Subject  Matter  Experts  and  analysts. 

2.  Able  to  use  past  experience  to  outline  the  information  flow/activities  for  a  given  scenario. 

3.  Task  analyses  and  data  can  be  reusable  between  missions/systems. 

4.  Able  to  identify  increased  workload,  and  task  conflicts. 

Disadvantages: 

1 .  Extensive  effort  to  complete  a  MFTA. 

2.  Individuals  conducting  the  MFTA  may  not  be  the  same  individuals  responsible  for  designing  the 
system,  creating  the  need  for  a  large  data  transfer. 

IAI  Applicability: 

1.  MFTA  task  hierarchy  would  need  to  be  modified  and  expanded  to  consider  intelligent  agents 
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(Chow  et.  at.,  2006). 

2.  Produces  information  and  action  requirements  which  could  inform  training  and  interface  design, 
or  task  re-allocation  to  support  multiple  operators/systems. 

3.  Able  to  identify  high  areas  of  workload  and  task  conflicts,  identifying  where  system  support  may 
be  required. 


s 

Chow,  R.,  Kobierski,  B.,  Coates,  C.  and  Crebolder,  J.  (2006).  Applied  comparison  between 
hierarchical  goal  analysis  and  mission  function  task  analysis.  Proceedings  of  the  50th  Human 
Factors  and  Ergonomics  Society  Conference,  San  Francisco,  CA 


Overview: 

This  paper  compares  the  application  of  Mission,  Function  and  Task  Analysis  and  Hierarchical  Goal 
Analysis  to  identify  requirements  for  systems  design  in  a  military  context.  The  two  approaches  were 
used  to  analyze  three  tactical  positions  in  the  Operations  Room  of  a  Halifax  Class  naval  frigate. 

The  first  application  used  HGA  to  support  the  design  of  intelligent,  adaptive  interfaces  for  UAV 
control  by  a  3-person,  airborne  crew.  The  second  application  used  HGA  to  identify  critical  activities 
that  could  benefit  from  advanced  decision  aiding  technology  in  the  operations  room  of  a  Halifax- 
class  naval  frigate. 

Findings: 

MFTA 

1 .  Relatively  easy  to  apply  by  analysts  and  subject  matter  experts,  who  had  little  difficulty 
identifying  actions  (i.e.,  tasks)  that  need  to  be  performed  to  support  a  given  function  or  that  need 
to  be  performed  in  parallel  or  in  sequence.  Most  analysts  and  SMEs  were  familiar  with  MFTA. 

2.  Task  sequences  were  constructed  bottom-up  from  task  lists;  they  supported  evaluation  of  the 
operators’  abilities  to  multi-task. 

3.  The  volume  of  effort  required  is  quite  extensive  (e.g.,  2600  tasks  to  analyze)  and  could  increase 
rapidly  if  more  operators  were  added  (cost). 

4.  More  difficult  to  transfer  requirements  to  design  recommendations  because  this  analysis 
produces  very  detailed  data. 

HGA 

1 .  Required  substantial  training,  to  convert  operators  from  thinking  about  what  actions  a  given 
operator  needs  to  perform,  to  thinking  about  who  needs  to  assess  effects  and  how  effects  need 
to  be  assessed. 

2.  Needed  to  be  worked  both  top  down  and  bottom-up  (reflecting  how  an  operator  may  direct 
attention),  and  the  endpoint  for  analysis  was  not  as  obvious.  SMEs  found  it  harder  to  review 
goal  hierarchies  than  task  sequences,  because  goals  in  the  same  part  of  a  hierarchy  were  not 
necessarily  aligned  or  related  in  time. 

3.  Operators  were  assigned  after  all  goals  were  identified,  so  consideration  of  all  operators  who 
might  interact  with  the  targeted  operators  was  already  “built  into”  the  analysis  (cost). 

4.  Transfer  of  requirements  to  design  recommendations  is  more  direct  as  the  HGA  database  is 
more  abstract  in  nature,  but  smaller  and  more  focused  in  that  it  does  not  attempt  to  track  a 
mission  timeline.  Information  to  be  displayed  and  controls  necessary  to  complete  specific  goals 
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are  clearly  indicated.  Also  included  are  other  control  requirements  (e.g.,  limit  access  to  a 
variable)  that  may  be  addressed  by  system  or  process  design,  and  shared  information 
requirements  (e.g.,  feedback  on  a  goal  that  affects  another  goal)  that  may  be  addressed  by 
interface  design  or  by  facilitating  communication  between  operators. 


Conclusions  for  IASs: 

1 .  MFTA  was  found  to  be  easy  to  learn  and  use  whereas  HGA  required  a  heavy  initial  investment 
in  terms  of  time  and  effort  to  learn,  and  required  continual  support  from  a  knowledgeable 
support  team  to  ensure  that  the  domain  experts’  efforts  were  meaningfully  applied. 

2.  MFTA  seemed  especially  suited  for  the  design  of  training  or  interfaces  for  specific  operators. 

3.  HGA  seemed  especially  suited  for  system-level  design,  such  as  the  design  of  a  new  operations 
room  involving  new  roles,  physical  layouts,  and  technological  support. 

4.  The  transfer  of  requirements  to  design  recommendations  was  more  direct  and  easier  for  HGA 
than  MFTA. 


4.2.3  Hierarchical  Goal  Analysis  based  on  the  Perceptual  Control 
Theory 


Hierarchical  Goal  Analysis  based  on  the  Perceptual 
Control  Theory 


Reference: 

Darvill,  D.,  Kumagai,  J.  &  Youngson,  G.  (2006).  Requirements  Analysis  Methods  for  Complex 
Systems.  Technical  Report  to  Defence  Research  &  Development  Canada  Valcartier  (CR  2005- 
076). 

Hendy,  K.,  Beevis,  D.,  Lichacz,  F.  &  Edwards,  J.  (2001).  Analysing  the  Cognitive  System  from  a 
Perceptual  Control  Theory  Point  of  View.  Defence  Research  and  Development  Canada  (TO) 
Report:  SL2001-143. 

Hou,  M.  &  Kobierski,  B.  (2005).  Performance  Modeling  of  Agent-Aided  Operator-Interface 
Interaction  for  the  Control  of  Multiple  UAVs.  2005  IEEE  International  Conference  on  Systems, 
Man,  and  Cybernetics. 

Kobierski,  B.  (2004).  Hierarchical  Goal  Analysis  and  Performance  Modelling  for  the  Control  of 
Multiple  UAVs/UCAVs  from  an  Airborne  Platform.  Contract  Report  to  Defence  Research  and 
Development  Canada  (CR  2004-063). 


Overview: 

The  Perceptual  Control  Theory  models  human-world  relationships  and  considers  the  human  as  a 
negative  feedback  loop  system,  interacting  with  an  environment  prone  to  disturbances.  This 
negative  feedback  system  is  error-correcting,  and  therefore,  the  human  exhibits  compensatory 
behaviours  to  achieve  system  stability.  Hendy  et.  al.,  (2001)  describe  the  PCT  model  as  a  multi¬ 
layered  system,  with  multiple  goals  providing  the  reference  points  for  a  hierarchical  organization  of 
control  loops.  These  loops  can  provide  control  at  many  levels,  including  the  lowest  levels  of 
sensory  processing,  to  higher-level,  more  abstract  goals. 
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Hendy  et.  al.,  (2001)  indicated  that: 

“In  PCT  terms,  an  emitted  action  or  behaviour  is  in  response  to  the  presence  of  an  error,  or 
difference,  signal.  The  emitted  action  transmitted  purposefully,  with  the  intention  of  changing 
the  state  of  the  world  so  that  the  Operator’s  perception  can  be  made  to  match  a  desired 
state  or  goal,  reducing  the  error  signal”.  “The  hierarchical  structure  of  goals  and  objectives, 
from  the  highest  level  of  abstraction  to  the  lowest,  represents  the  hierarchy  of  control  loops 
that  potentially  will  be  active  during  the  life  of  the  system.  Any  goal/objective  not  served  by  a 
control  loop  has  no  influence  over  a  variable  in  the  external  world,  and  will  cause  no 
behaviour/output  to  be  emitted.  Alternatively,  all  system  variables  that  are  to  be  influenced 
must  be  associated  with  a  goal  or  objective”. 

Hierarchical  Goal  Analysis  combines  function  and  task  analysis  into  a  signal  process.  HGA  is 
rooted  in  PCT,  and  therefore  is  based  on  the  idea  that  humans  and  machine  can  be  described  in 
terms  of  a  hierarchical  control  model  (Hou  &  Kobierski,  2005).  Furthermore,  HGA  emphasizes  that 
goal  directed  human  activity  is  driven  by  a  process  of  a  closed  loop  negative  feedback  control. 

HGA  claims  that  all  human  behaviour  occurs  as  a  result  of  a  perceptually  driven,  goal-referenced 
feedback  system.  This  approach  acknowledges  the  goals  and  knowledge  of  humans  in  the  system, 
but  is  equally  concerned  with  the  sensory,  perceptual  and  psychomotor  requirements. 

Process: 

HGA  starts  with  a  goal  at  its  highest  level  of  the  hierarchy.  Analysing  the  goal  in  a  top-down 
framework,  the  HGA  requires  a  decomposition  of  the  goal  from  its  highest  level,  down  to  its  lowest 
levels.  From  a  PCT  perspective,  goals  are  divided  in  perceptual  terms. 

Hendy  et.  al.,  (2001)  provides  the  following  rules  for  conducting  a  PCT-based  HGA: 

•  All  points  are  generalized  goals/objectives  until  assigned  to  a  human  or  machine. 

Therefore,  goals  assigned  to  humans  are  perceptual  goals  that  drive  human  activity.  Any 
goal  that  is  not  assigned  to  a  human  or  machine  is  not  actively  controlling  (i.e.,  this  type  of 
goal  is  likely  not  to  be  achieved  as  it  is  not  assigned  to  any  agent).  All  PCT  goal 
statements  are  in  the  form  of  “I  want  to  perceive . ”. 

•  All  control  loops  involve  a  variable  that  in  influenced/controlled  by  the  loop  action.  Each 
goal/objective  must  have  an  influenced  variable.  If  a  human  is  assigned  to  a 
goal/objective,  the  variable  can  be  internal  or  external. 

•  Moving  downwards  through  the  hierarchy  creates  sub-goals/sub-objectives.  The 
decomposition  into  sub-goals  and  sub-objectives  follows  a  means-end  hierarchy. 

PCT-based  HGA  requires  specific  cognitive  and  perceptual  information  related  to  each  goal  loop  at 
each  hierarchical  level,  such  as  (Darvill  et.  al.,  2006): 

•  Required  Knowledge  States,  both  declarative  knowledge  and  situational  knowledge  are 
identified; 

•  Initiating  Conditions,  when  the  task  begins; 

•  Ending  Conditions.  When  the  task  is  considered  complete  (to  move  to  the  next  task); 

•  Perceptual/Cognitive  Processes,  processes  associated  with  the  goal  chosen  from  a  list 
relevant  to  the  current  goal; 

•  Inputs/Sensations,  inputs  and  sensations  chosen  from  a  list,  such  as:  for  memory  -  verbal 
recall  required;  for  visual  -  text  is  required); 

•  Outputs/Behaviours,  outputs  and  behaviours  chosen  from  a  list,  such  as:  for  voice  - 
establish  radio  link;  and 
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•  Multiple  Agents.  Multiple  agents  may  be  interacting  through  their  influence  on  shared 
environmental  variables. 

The  assignment  of  objectives  is  a  major  engineering  decision  that  fundamentally  shapes  the  to-be- 
designed  system  (Hendy  et.  al.,  2001),  so  no  assignments  to  human  or  machine  are  made  using 
HGA  until  the  goal-related  information  has  been  collected  for  all  system  goals  within  the  hierarchy 
(Darvill  et.  al.,  2006). 


Advantages: 

1 .  PCT-based  HGA  addresses  many  of  the  deficiencies  associated  with  traditional  HGA  (Hendy 
et.  al.,  2001). 

2.  PCT-based  HGA  provides  an  additional  step  to  Hierarchical  Task  Analysis  such  that  it 
acknowledges  the  need  for  error  correction  at  all  levels  within  the  hierarchy. 

3.  HGA  can  be  integrated  into  the  engineering  design  process. 

4.  Considers  all  goals  (highest-level  to  lowest-level)  as  possibilities  to  be  assigned  to  agents, 
either  human  or  machine. 

Disadvantages: 

1 .  Considerable  time. 

2.  Training  and  experience  required  to  implement. 

IAI  Applicability: 

1 .  Designing  for  control  loop  stability  between  an  Operator  and  machine  can  ensure  that  the 
proper  controls  and  displays  are  embedded  in  an  interface,  to  ensure  that  perceptual  errors 
inherent  in  human-machine  interaction  are  minimized. 

2.  The  HGA  hierarchy  accounts  for  error  correction  at  all  levels  of  the  goal  hierarchy. 

3.  The  primary  output  from  a  HGA  analysis  is  a  goal  structure  which  provides  interface  guidance 

4.  Output  from  a  HGA  analysis  will  identify  cognitive  and  perceptual  information  related  to  Output 
Interfaces  and  Input  Interfaces. 


Hou,  M.  and  Kobierski,  R.D.  (2006)  Operational  Analysis  and  Performance  Modeling  for  the 
Control  of  Multiple  Uninhabited  Aerial  Vehicles  from  an  Airborne  Platform.  In  Advances  in  Human 
Performance  and  Cognitive  Engineering  Research,  7,  267-282.  Elsevier 


Overview: 

Detailed  are  the  results  from  an  operational  analysis  and  simulation  of  the  first  phase  of  the  DRDC 
project  to  investigate  the  efficacy  of  lAls  in  an  operational  situation.  The  simulation  was  based  on 
CF  maritime  aircraft  operations  in  support  of  counter-terrorism  activities. 

Process  of  analysis: 

A  Hierarchical  Goal  Analysis  was  first  performed  to  gain  a  more  detailed  understanding  of  the 
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goals  and  tasks  involved  in  controlling  multiple  UAVs  from  an  airborne  platform  (CP140  aircraft). 
Methods  of  analysis  are  based  on  Perceptual  Control  Theory. 

Operations/missions  were  analyzed  as  hierarchies  of  goals.  These  goals  “are  nested,  sequenced 
or  linked  into  logical  networks”.  The  result  of  this  analysis  produced  an  HGA  and  network  model  (in 
IPME). 

These  network  models  are  then  used  to  model  and  predict  operators’  performance  (assignment  of 
operators  to  tasks,  and  interactions).  The  goals  and  tasks  identified  by  HGA  can  then  be  used  to 
determine  which  tasks  should  be  automated. 


Conclusions  for  IASs: 

1 .  The  analysis  revealed  HGA  can  provide  effective  task  models  that  can  be  used  to  improve  the 
operations  of  a  UAV  crew  with  IAI. 


4.2.4  Goal  Directed  Task  Analysis 


Goal  Directed  Task  Analysis 


Reference: 

Jones,  D.G.  &  Endsley,  M.R.  (2005).  Goad  Directed  Task  Analysis.  In  Hoffman,  R.,  Protocols  for 
Cognitive  Task  Analysis.  State  of  Florida  Institute  for  Human  and  Machine  Cognition. 

Endsley,  M.R.,  Bolstad,  C.A.,  Jones,  D.G.  &  Riley,  J.M.  (2003).  Situation  Awareness  Oriented 
Design:  From  User’s  Cognitive  Requirements  to  Creating  Effective  Supporting  Technologies. 
Proceedings  of  the  Human  Factors  and  Ergonomics  Society  47th  Annual  Meeting.  Denver, 
Colorado. 

Bolstad,  C.A.,  Riley,  J.M.,  Jones,  D.G.  &  Endsley,  M.R.  (2002).  Using  Goal  Directed  Task  Analysis 
with  Army  Brigade  Officer  Teams.  Proceedings  of  the  Human  Factors  and  Ergonomics  Society 
46th  Annual  Meeting.  Baltimore,  MD. 

Endsley,  M.  R.,  Bolte,  B.  &  Jone,  D.  G.  (2003).  Designing  for  Situation  Awareness.  New  York: 
Taylor  and  Francis 


Overview: 

A  GDTA  identifies  the  goals  that  must  be  achieved  to  accomplish  a  mission,  the  decisions  that 
must  be  made  to  accomplish  the  goals,  and  the  information  that  is  required  to  support  these 
decisions.  Specifically,  a  GDTA  outlines  the  information  required  to  perform  a  job,  and  details  the 
integration/combination  of  this  information  to  formulate  a  decision.  The  process  focuses  on  the 
information  required  to  meet  each  goal;  this  process  however  does  not  focus  on  the  means  in 
which  an  Operator  acquires  the  information. 

A  GDTA  also  defines  the  Situation  Awareness  requirements  required  to  assess  the  information  to 
determine  how  to  achieve  each  goal  for  successful  task  completion.  Once  the  information  required 
to  achieve  a  goal  is  identified,  an  evaluation  of  a  pre-existing  system  can  be  conducted  to 
determine  whether  the  current  design  satisfies  these  needs,  or  a  future  system  can  be  designed  to 
account  for  these  needs  prior  to  system  development. 
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Process: 

A  GDTA  has  three  main  components,  including: 

•  Goals.  Represent  “higher-order”  objectives  related  to  a  task,  mission,  or  operation  that  are 
essential  for  successful  job  performance.  The  highest  level  goal  represents  the  “overall  goal”  of 
the  decision  maker.  Each  main  goal  will  have  a  varying  number  of  sub-goals.  Goals  represent 
the  cognitive  effort  required  for  successful  task  completion. 

•  Decisions.  Represents  the  decisions  that  must  be  made  to  achieve  a  particular  goal.  Decisions 
represent  the  questions  the  decision  maker  must  answer  in  order  to  achieve  a  goal. 

•  Situation  Awareness  Requirements.  Represents  the  information  required  to  answer  the 
questions  that  form  the  basis  of  decisions. 

A  GDTA  is  documented  according  to  two  main  structures: 

•  Goal  Hierarchy.  The  overall  goal  is  identified,  from  which  the  major  goals  are  defined,  along 
with  the  sub-goals  required  to  successfully  achieve  each  major  goal. 

•  Relational  Hierarchy.  Outlines  the  relationship  between  the  goals,  the  sub-goals,  the  decisions 
relevant  to  each  sub-goal,  and  the  SA  requirements  relevant  for  each  decision. 

Advantages: 

1 .  Details  the  situational  awareness  requirements  relevant  to  attaining  each  goal;  and 

2.  Identification  of  the  SA  requirements  will  aid  evaluation  and  the  design  of  systems  to  ensure 
that  the  system  supports  an  Operator  in  building  and  maintaining  a  high-level  of  SA. 

Disadvantages: 

1 .  Comprehensive  method  taking  extensive  time  to  complete; 

2.  Requires  several  sessions  with  subject  matter  experts  to  define  the  domain;  and 

3.  Degree  of  subjectivity  during  the  SME  sessions. 

IAI  Applicability: 

Aids  the  design  of  systems  to  support  SA,  by: 

1 .  Identifying  what  information  an  Operator  needs  to  know,  providing  guidance  for  designing  a 
meaningful  interface  design; 

2.  Identifying  functional  grouping  of  information; 

3.  Guiding  the  relationship  between  information  and  decisions  to  support  goals;  and 

4.  Identifying  critical  cues  required  to  direct  shifts  in  task  priority. 


4.2.5  Cognitive  Task  Analysis 

Cognitive  Task  Analysis 


Reference: 

Zachary,  W.,  Ryder,  J.  &  Hicinbothom,  J.  (n.d.).  Cognitive  Task  Analysis  and  Modeling  of  Decision 
Making  in  Complex  Environments.  CHI  Systems  Incorporated,  Lower  Gwynedd,  PA 
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Clark,  R.  &  Estes,  F.  (1996).  Cognitive  Task  Analysis.  International  Journal  of  Educational 
Research.  25(5),  403-417. 

Clark,  R.,  Feldon,  D,  van  Merrienboer,  J.,  Yates,  K.  &  Early,  S.  (2006).  Cognitive  Task  Analysis. 
Accessed  at: 

http://www.coqtech.usc.edu/publications/clark  etal  cognitive  task  analysis  chapter.pdf.  January 

27,  2007. 

Potter,  S.,  Roth,  E.,  Woods,  D.  &  Elm,  W.  (2000).  Bootstrapping  Multiple  Converging  Cognitive 
Task  Analysis  Techniques  for  System  Design.  In  Schraagen,  J.M.C.,  Chipman,  S.F.,  &  Shalin,  V. 
L.  (Eds.),  Cognitive  Task  Analysis.  Mahwah,  NJ:  Lawrence  Erlbaum  Associates. 


Overview: 

Cognitive  Task  Analysis  captures  the  experience,  knowledge,  and  intuition  of  Subject  Matters 
Experts  by  uncovering  the  cognitive  skills  and  abilities  required  to  complete  a  task  proficiently.  CTA 
concentrates  on  elements  that  cannot  be  directly  observed  such  as  difficult  decisions,  judgments, 
and  perceptual  skills.  CTA  identifies  the  demands  that  are  placed  on  an  operator’s  cognitive 
resources  such  as  memory,  attention,  and  decision  making.  Note  that  cognitive  task  analysis  does 
not  replace  a  behavioural  task  analysis,  but  rather  supplements  it.  Concentrating  on  these 
elements  provides  a  means  to  examine  the  processes  underlying  behaviour.  This  information 
allows  a  designer  to  focus  on  the  system  features  that  an  operator  will  find  most  difficult  to  learn 
and  therefore,  the  features  that  will  likely  create  error. 

Another  characteristic  of  CTA  is  an  attempt  to  describe  the  differences  between  novices  and 
experts  in  the  development  of  knowledge  about  tasks.  CTA  is  able  to  elicit  mental  models  used  by 
experts,  which  provides  a  good  basis  for  design  and  training. 

CTA  encompasses  a  collection  of  diverse  approaches  with  very  little  connection  or  cohesiveness 
(Potter  et.  al.,  2000).  However,  all  CTA  approaches  share  a  common  goal:  to  identify  the  cognitive 
activities  that  underlie  task  performance  in  order  to  improve  individual  and  team  performance 
(through  correct  design  of  training  aids,  OMIs,  or  decision  aids)  (Potter  et.  al.,  2000). 

Process: 

CTA  can  be  decomposed  into  three  phases: 

•  Knowledge  Elicitation.  Extraction  of  information  through  in-depth  interviews  and 
observations,  about  cognitive  events,  structures  or  models.  Interviews  are 
conducted  with  Subject  Matter  Experts. 

•  Data  Analysis.  CTA  practitioners  use  a  variety  of  quantitative  and  qualitative 
analyses  to  complete  data  analysis.  The  goal  is  to  simplify,  abstract,  and  transform 
the  data  to  develop  explanations  and  meaning. 

•  Knowledge  Representation.  Process  of  displaying  the  data  and  depicting  the 
relationship,  explanations,  and  the  meaning  derived  from  data  analysis. 


Advantages: 

1 .  CTA  can  boost  human  performance  by  guiding  the  development  of  tools  and  programs  that 
support  the  cognitive  processes  required  for  a  task. 

2.  Able  to  analyze  task  performance  in  situations  that  involve  change,  uncertainty  and  time 
pressure. 

3.  Aid  experts  in  articulating  knowledge  that  is  generally  difficult  to  verbalize. 
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Disadvantages: 

1 .  CTA  encompasses  a  collection  of  diverse  approaches  with  very  little  connection  or 
cohesiveness  (Potter  et.  al.,  2000). 

2.  CTA  cannot  be  viewed  as  a  standalone  analysis.  It  needs  to  be  an  iterative  process  that  learns 
from  subsequent  design  activities  (Potter  et.  al.,  2000). 

IAI  Applicability: 

1 .  CTA  can  boost  human  performance  by  guiding  the  development  of  tools  and  programs  that 
support  the  cognitive  processes  required  for  a  task. 

2.  CTA  must  work  within  a  system  development  process  and  support  critical  system  design 
issues  (Potter  et.  al.,  2000). 


4.2.6  Team  Cognitive  Task  Analysis 


Team  Cognitive  Task  Analysis 


Reference: 

Baker,  D.P.,  Salas,  E.  &  Cannon-Bowers,  J.A.  (n.d.).  Team  Task  Analysis:  Lost  But  Hopefully  Not 
Forgotten.  Accessed  at:  http://www.air.org/teams/publications/teamwork/team  task  analvsis.pdf. 
January  27,  2007. 

Blickensderfer,  E.,  Cannon-Bowers,  J.  A.,  Salas,  E.,  &  Baker,  D.  P.  (2000).  Analyzing  Knowledge 
Requirements  in  Team  Tasks,  In  Schraagen,  Chipman,  &  Shalin,  Eds.  Cognitive  Task  Analysis. 
Mahwah,  NJ:  Lawrence  Erlbaum.  431-447. 

Bowers,  J.,  Baker,  D.P.,  &  Salas,  E.  (1994).  Measuring  the  Importance  of  Teamwork:  The 
Reliability  and  Validity  of  Job/Task  Analysis  Indices  for  Team-Training  Design.  Military  Psychology, 
6(4),  205-214. 

Brenner,  T.,  Sheehan,  K.,  Arthur,  W.  &  Bennett,  W.  (n.d.).  Behavioural  and  Cognitive  Task 
Analysis  Integration  for  Assessing  Individual  and  Team  Work  Activities.  Accessed  at: 
http://www.internationalmta.org/1998/9847d.html.  on  January  27,  2007. 

Darvill,  D.,  Kumagai,  J.  &  Youngson,  G.  (2006).  Requirements  Analysis  Methods  for  Complex 
Systems.  Technical  Report  to  Defence  Research  &  Development  Canada  Valcartier  (CR  2005- 
076). 

Harder,  R.  &  Higley,  H.  (2004).  Application  of  Thinklets  to  Team  Cognitive  Task  Analysis. 
Proceedings  of  the  37th  Hawaii  International  Conference  on  System  Sciences. 


Overview: 

Team  Cognitive  Task  Analysis  is  an  extension  of  Cognitive  Task  Analysis,  with  emphasis  on 
teamwork  requirements.  Current  methods  of  task  analysis  fail  to  capture  team  characteristics  such 
as  interdependence  and  co-operation.  Applying  a  method  of  analysis  designed  for  individuals  to 
teams  is  not  sufficient  for  obtaining  true  understanding  of  how  a  team  works. 

Team  CTA  methods  view  the  team  as  an  intelligent  entity,  and  attempt  to  identify  the  cognitive 
processes  required  by  team  dependent  tasks  (Harder  &  Higley,  2004).  Team  CTA  captures  the 
cognitive  processes  of  a  team,  and  focuses  on  the  way  a  team  coordinates  the  understanding  of 
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the  different  members  and  synthesizes  the  task  elements  (Harder  &  Higley,  2004).  This  method 
emphasizes  the  importance  of  communication  and  situation  awareness  and  provides  assistance  in 
diagnosing  and  treating  existing  problems  in  teamwork  to  ensure  efficient  and  effective  team 
functioning. 

Team  CTA  provides  the  foundation  for  many  human  resource  functions  in  the  context  of  teamwork 
including  team  task  design,  team  composition,  team  training  and  team  composition  (Brenner  et. 
al.).  Team  CTA  also  provides  insight  into  critical  team  task  elements. 

Team  task  analysis  refers  not  only  to  an  analysis  of  a  team’s  tasks,  but  also  to  a  comprehensive 
assessment  of  a  team’s  teamwork  requirements  (i.e.,  knowledge,  skill,  ability,  and  attitude 
requirements).  Team  task  analysis  is  important  because  it  forms  the  foundation  for  team  design, 
team  performance  measurement,  and  team  training.  Essentially,  it  is  the  building  block  for  all 
"team"  resource  management  functions  (Baker  et.  al.). 

Darvill  et.  al.,  (2006)  identifies  a  list  of  cognitive  processes  that  are  important  in  the  analysis  of 
teams,  including: 

•  Control  of  attention; 

•  Shared  situation  awareness; 

•  Shared  mental  models; 

•  Application  of  strategies  and  heuristics  to  make  decisions;  and, 

•  Meta-cognition  or  how  a  team  is  able  to  monitor  itself  and  determine  when  the  team  is  faced 
with  difficulties  or  challenges. 

Process: 

Although  there  is  much  research  invested  in  the  application  of  Task  Analysis  techniques,  there  is 
relatively  little  guidance  in  literature  regarding  how  to  conduct  task  analyses  for  teams  (Bowers  et. 
al.,  1994). 

Currently,  the  primary  method  for  conducting  team  cognitive  task  analysis  has  been  to  use 
techniques  from  job  analysis  to  determine  team  task  and  cognitive  skill  requirements  (Baker  et. 
al.).  A  literature  review  also  conducted  by  Baker  et.  al.,  (n.d.)  provided  examples  of  the  following 
team  cognitive  task  analysis  methods:  Critical  Incident  Technique,  Task  Important  Indices,  Team 
Task  Inventory,  and  Team  Characteristic  Questionnaires.  Despite  the  Team  CTA  technique  used, 
the  results  of  the  assessment  should  provide  a  comprehensive  evaluation  of  a  team,  providing 
information  on  the  tasks  performed  in  terms  of  communication,  cooperation,  knowledge,  attitude 
and  skill  requirements. 

Darvill  et.  al.,  (2006)  also  indicates  that  Team  CTA  is  lacking  support  in  literature.  However,  the 
following  methods  to  Team  CTA  were  identified: 

•  Team  Audit.  Elicits  knowledge  and  skills  from  team  members,  and  elicits  examples  from  actual 
events. 

•  Team  Critical  Decision  Method  (Team  CDM).  Four  information  gathering  sweeps  are 
conducted  to  reveal  critical  cognitive  elements  related  to  team  incidents. 

•  Distributed  Team  Assessment  Method  (DTAM.  Method  used  with  a  widely  separated  team  that 
was  involved  in  the  same  incident.  The  objective  of  this  method  is  to  identify  functions,  goals, 
and  communication  links  in  order  to  reveal  overlaps  or  gaps  in  roles  and  functions.  This  will 
identify  goal  conflicts  and  will  provide  the  opportunity  to  improve  information  exchange. 

•  Decision  Requirements  Exercise  (DRE).  Decision  requirements  table  to  determine  the  critical 
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decisions  and  judgements  a  team  made  to  perform  a  task. 

•  Wagon  Wheel  Method  (WWM).  Snapshot  of  team  communication  links  and  the  nature  of  their 
communications. 

Advantages: 

1 .  The  output  of  Team  CTA  provides  input  to  team  design,  team  performance  measurement,  and 
team  training. 

2.  The  output  of  Team  CTA  provides  results  that  can  act  as  input  to  other  analysis  methods. 

3.  Useful  for  analysis  of  complex  multi-person  judgements  and  decision-making. 

4.  Aids  the  design  of  systems  and  interfaces  that  are  used  for  teams. 

Disadvantages: 

1 .  Extensive  time  and  expertise  required. 

2.  Common  challenges  are  apparent  when  conducting  group  interviews. 

3.  Little  research  regarding  the  application  of  Team  CTA  techniques. 

4.  Application  of  an  analysis  method  to  teams  designed  for  individuals  is  not  sufficient  for  true 
understanding  of  how  a  team  works. 

IAI  Applicability: 

1 .  Views  a  team  as  an  intelligent  entity,  and  attempts  to  identify  the  cognitive  processes  required 
by  team  dependent  tasks. 

2.  Captures  the  cognitive  processes  of  a  team,  and  focuses  on  the  way  a  team  coordinates  the 


understanding  of  the  different  members  and  the  synthesis  of  task  elements. 

3.  Used  to  identify  information,  cues,  and  strategies  required  to  make  key  decisions. 


4.2.7  Applied  Cognitive  Task  Analysis 

Applied  Cognitive  Task  Analysis 

Reference: 

Eddy,  M.,  Kribs,  H.  &  Cowen,  M.  (1999).  Cognitive  and  Behavioural  Task  Implications  for  Three- 
Dimensional  Displays  used  in  Combat  Information/Direction  Centers.  Technical  Report  1792. 

Militello,  L.,  Hutton,  R.,  Pliske,  R.,  Knight,  B.  &  Klein,  G.  (1997).  Applied  Cognitive  Task  Analysis 
Methodology.  Navy  Personnel  Research  and  Development  Center.  San  Diego,  CA. 

Militello,  L.  &  Hutton,  R.  (1998).  Applied  Cognitive  Task  Analysis:  A  Practitioner’s  Toolkit  for 
Understanding  Cognitive  Task  Demands.  Ergonomics  41(11),  1618-1641. 

Overview: 

ACTA  is  a  streamlined  method  of  Cognitive  Task  Analysis.  The  basis  of  ACTA’s  development  was 
to  develop  techniques  that  would  enable  systems  and  instructional  designers  to  elicit  critical 
cognitive  task  components  from  Subject  Matter  Experts.  The  ACTA  was  developed  to  allow  cues 
and  sources  of  information  to  be  derived  within  the  context  of  Situation  Awareness.  ACTA 
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analyses  an  individual’s  mental  representation  of  SA,  and  compares  the  differences  between 
novice  and  expert  processes  (Eddy  et.  al.,  1999). 

Process: 

ACTA  consists  of  three  interview  methods  that  help  the  practitioner  to  extract  information  about  the 
cognitive  demands  and  skills  required  for  a  task. 

•  Task  Diagram.  The  Task  Diagram  provides  an  overview  of  a  task  and  identifies  a  task’s 
cognitive  elements.  The  development  of  a  Task  Diagram  is  facilitated  through  a  preliminary 
interview,  providing  a  surface-level  view  of  the  task’s  cognitive  elements.  This  allows  the 
interviewer  to  identify  the  most  difficult  and  relevant  cognitive  elements  that  should  be  the 
focus  of  interviews  during  the  Knowledge  Audit  and  Simulation  Interview  stage.  The  Subject 
Matter  Experts  decompose  a  task  into  sub-tasks.  After  the  sub-tasks  have  been  identified,  the 
SME  identifies  the  sub-tasks  that  require  cognitive  skill. 

•  Knowledge  Audit.  The  Knowledge  Audit  captures  the  most  important  aspects  of  expertise.  The 
Audit  identifies  how  expertise  is  used,  and  provides  examples  based  on  true  experience.  The 
Audit  is  based  on  knowledge  categories  that  form  the  basis  of  expertise,  such  as:  diagnosing 
and  predicting,  situation  awareness,  perceptual  skills,  improvising,  metacognition,  recognizing 
anomalies,  and  compensating  for  equipment  limitations  (Militello  et.  al.,  1997).  The  interviewer 
uses  a  set  of  probes  that  are  designed  to  describe  types  of  domain  knowledge  or  skill.  The 
goal  is  to  use  these  probes  to  determine  the  nature  of  these  expertise  skills,  the  tasks  where 
these  skills  are  implemented,  along  with  what  types  of  strategies  are  used.  The  output  of  the 
Knowledge  Audit  is  a  table  which  provides  an  inventory  of  task-specific  expertise.  The  table 
outlines  when  experience,  expertise,  and  strategies  were  required  for  difficult  situations,  and 
why  these  situations  may  pose  a  challenge  to  less-experienced  Operators  (Militello  et.  al., 
1997). 

•  Simulation  Interview.  The  purpose  of  the  Simulation  Interview  is  to  identify  a  SME’s  cognitive 
processes  within  the  context  of  an  incident.  The  interview  presents  a  challenging  scenario  to 
the  SME,  and  the  SME  is  required  to  identify  major  events,  judgment  and  decision  making 
requirements,  troubleshooting  diagnosis,  situation  assessment,  critical  cues,  and  selection  of 
courses  of  action. 


Advantages: 

1 .  ACTA  techniques  are  easy  to  use,  flexible,  and  provide  clear  output. 

2.  Identifies  where  a  system’s  design  must  support  human  problem-solving  and  decision-making 
by  assessing  complex  tasks  that  require  a  high  degree  of  cognitive  skill. 

Disadvantages: 

1 .  Although  ACTA  elicits  important  cognitive  information,  there  is  a  trade-off  when  using  a 
streamlined  approach;  the  more  streamlined  and  proceduralized  CTA  techniques  become,  the 
less  powerful  they  are  (Militello  &  Hutton,  1998). 

2.  ACTA  techniques  may  gather  less  comprehensive  information  than  more  systematic 
techniques. 

IAI  Applicability: 

1 .  ACTA  provides  data  that  translates  more  directly  into  applied  products  such  as  improved 
training  scenarios  or  interface  recommendations. 

2.  Allows  systems  designers  to  elicit  and  represent  critical  cognitive  components  of  skilled  task 
performance,  and  the  means  to  transform  these  data  into  design  recommendations. 
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3.  ACTA  techniques  were  developed  to  elicit  critical  cognitive  task  components  from  Subject 
Matter  Experts. 


4.2.8  Cognitive  Work  Analysis 


Cognitive  Work  Analysis 


Reference: 

Darvill,  D.,  Kumagai,  J.  &  Youngson,  G.  (2006).  Requirements  Analysis  Methods  for  Complex 
Systems.  Technical  Report  to  Defence  Research  &  Development  Canada  Valcartier  (CR  2005- 
076). 

Fidel,  R.  &  Pejtersen,  A.  (2004a).  Cognitive  Work  Analysis. 

Fidel,  R.  &  Pejtersen,  A.  (2004b).  From  information  behaviour  research  to  the  deign  of  information 
systems:  the  Cognitive  Work  Analysis  Framework.  Information  Research,  10(1). 

Lui,  F.  &  Watson,  M.  (2002).  Mapping  Cognitive  Work  Analysis  (CWA)  to  an  Intelligent  Agents 
Software  Architecture:  Command  Agents.  Proceedings  of  the  Defence  Human  Factors  Special 
Interest  Group  (DHFSIG).  DSTO  Melbourne,  Australia. 

Naikar,  N.  (2006).  An  Examination  of  the  Key  Concepts  of  the  Five  Phases  of  Cognitive  Work 
Analysis  with  Examples  from  a  Familiar  System.  Proceedings  of  the  Human  Factors  and 
Ergonomics  Society  50th  Annual  Meeting.  San  Francisco,  CA. 

Rasmussen,  J.,  Pejtersen,  A.  &  Schmidt,  K.  (1990).  Taxonomy  for  Cognitive  Work  Analysis.  RISO 
National  Laboratory,  Cognitive  Systems  Group,  Denmark. 


Overview: 

CWA  is  useful  for  the  study  of  human-information  interaction  and  for  the  design  of  information 
systems  and  services.  The  approach  analyzes  the  work  individuals  perform,  the  tasks  they 
perform,  the  decisions  that  they  make,  their  information  behaviour,  and  the  context  in  which  their 
work  is  performed.  The  purpose  of  this  approach  is  to  facilitate  systems  design,  and  facilitate  the 
analysis  of  tasks  and  context  simultaneously. 

CWA  is  a  work-centred  formative  approach  to  work  analysis  which  focuses  on  how  work  can  be 
performed.  The  approach  recognizes  that  workers  have  many  options  in  terms  of  what  work  to 
perform,  when  to  perform  the  work,  and  how  to  perform  the  work.  (Naikar,  2006).  Thus,  CWA 
identifies  the  constraints  that  shape  the  work  and  information  behaviour. 

CWA  considers  people  who  interact  with  information  involved  in  their  work-related  activities  (i.e., 
ecological  aspects),  rather  than  “users”  of  systems.  CWA  views  human-information  interaction  in 
the  context  of  human  work  activities.  Fidel  &  Pejtersen  (n.d.)  indicate  that  in  order  to  design 
systems  that  work  harmoniously  with  humans,  the  following  must  be  defined: 

•  The  work  individuals  perform; 

•  The  individuals’  information  behaviour; 

•  The  context  in  which  the  individuals  work;  and, 

•  The  reasons  for  their  actions. 

Process: 
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CWA  consists  of  the  following  five  stages. 

•  Work  Domain  Analysis  (WDA).  Focuses  on  the  purposive  (reasons  for  which  a  system 
exists)  and  physical  (resources  that  are  available)  environments  in  which  workers  operate 
(Naikar,  2006).  The  main  modeling  tool  is  the  abstraction-decomposition  space  (ADS). 

•  Control  Task  Analysis  (ConTA).  This  stage  focuses  on  what  needs  to  be  done  in  a  work 
domain.  The  ConTA  identifies  the  activity  that  is  necessary  to  achieve  the  objectives  of  a 
system  with  a  given  set  of  physical  resources.  According  to  Naikar  (2006),  ConTA  has 
three  key  concepts: 

•  Recognizes  that  the  same  goal  can  be  accomplished  in  different  ways  depending 
on  the  situation; 

•  An  activity  can  be  characterized  as  a  set  of  work  situations  or  work  functions;  and, 

•  An  activity  can  be  further  characterised  according  to  decision  making  functions  or 
control  tasks. 

The  tool  used  during  this  stage  is  a  decision  ladder. 

•  Strategies  Analysis.  This  stage  identifies  different  strategies  for  accomplishing  an  activity; 
thus,  Strategies  Analysis  is  focussed  on  how  an  activity  can  be  completed.  According  to 
Naikar  (2006)  Strategies  Analysis  has  four  key  concepts: 

•  Concerned  with  general  categories  of  cognitive  procedures; 

•  Several  strategies  are  often  available  to  complete  an  activity; 

•  Workers  will  often  utilize  more  than  one  strategy  when  completing  an  activity;  and 

•  The  range  of  strategies  that  are  possible  should  be  identified  as  opposed  to  only 
the  range  of  strategies  that  are  used. 

•  Social  Organization  and  Cooperation  Analysis  (SOCA).  Identifies  who  completes  the  work, 
and  how  the  work  is  shared  and  coordinated.  According  to  Naikar  (2006)  SOCA  has  four 
key  concepts: 

•  Flexible  organizational  structure  that  can  be  adapted  to  local  contingencies  are 
essential  for  dealing  with  unanticipated  events; 

•  Examines  how  the  work  demands  of  a  system  may  be  distributed  across  workers 
as  a  result  of  applying  various  criteria; 

•  Concerned  with  the  form  of  communication  or  social  organization  that  may  be 
adopted  for  coordination  in  a  system;  and 

•  Organizational  structures  in  many  systems  are  generated  in  real  time  by  multiple 
workers  responding  to  a  local  context. 

•  Worker  Competencies  Analysis  (WCA).  Identifies  the  competencies  that  a  worker  is 
required  to  adapt  to  the  work  requirements  of  a  system.  According  to  Naikar  (2006)  SOCA 
has  five  key  concepts: 

•  The  competencies  that  a  worker  requires  is  based  on  the  requirements  identified 
during  Stages  1-5; 

•  There  are  three  levels  of  cognitive  control  that  a  worker  can  use  to  perform  an 
activity:  skill-based,  rule-based,  and  knowledge  based; 

_ •  The  level  of  cognitive  control  that  is  implemented  depends  on  how  the  worker 
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interprets  the  information  in  the  environment;  and 

•  The  level  of  cognitive  control  that  is  implemented  also  depends  on  how  the 
information  is  presented  to  a  worker;  and 

•  Workers  will  implement  lower  levels  of  cognitive  control  more  quickly,  effectively, 
and  effortlessly  than  higher  levels  of  cognitive  control. 


Advantages: 

1 .  Results  from  CWA  can  be  transferred  directly  to  design  requirements. 

2.  Accounts  for  the  role  of  the  workers  in  complex  systems. 

3.  Focuses  on  analysing  the  environment. 

4.  The  model  provides  traceability  of  decision  making  in  a  organisational  structure  (Lui  &  Watson, 
2002) 

5.  The  model  provides  a  link  between  the  abstract  functions  in  the  higher  hierarchy  level  and  the 
plans/courses  of  action  in  the  lower  hierarchy  level  (Lui  &  Watson,  2002). 

Disadvantages: 

1 .  Complex  method  requiring  considerable  expertise. 

2.  Extensive  time  required  to  learn  and  use. 

3.  Difficult  to  define  and  map  the  system  on  all  five  stages. 

4.  Little  practical  difference  between  CWA  and  ACWA.  Many  times,  CWA  will  only  be  applied 
using  the  first  few  stages,  which  resembles  more  of  the  ACWA  process. 

5.  CWA  is  more  of  an  academic  endeavour  with  more  attention  being  placed  on  completing  the 
process  as  opposed  to  using  the  analysis  to  drive  design  and  develop  design  concepts  (Darvill 
et.  al„  2006). 

IAI  Applicability: 

1 .  Identifies  the  constraints  on  information  seeking,  including  the  individual  resources  and  the 
external  environment. 

2.  CWA  investigates  the  information  behaviour  in  context.  Therefore  the  results  are  valid  for  the 
design  of  information  systems  in  the  context  investigated,  rather  then  for  the  design  of  general 
information  systems  (Fidel  &  Pejtersen  (n.d.). 

3.  The  framework  facilitates  an  in-depth  examination  of  the  various  dimensions  of  a  context.  A 
study  of  a  particular  context  is,  therefore,  a  multi-disciplinary  examination  with  the  purpose  of 
understanding  the  interaction  between  people  and  information  in  the  work  context  (Fidel,  R.  & 
Pejtersen,  A.  (n.d.). 

4.  Provides  a  structure  of  human-information  interaction  analysis,  rather  than  subscribing  to 
specific  theories  or  models  (Fidel  &  Pejtersen  (n.d.). 

5.  Workers  will  implement  lower  levels  of  cognitive  control  more  quickly,  effectively,  and 
effortlessly  than  higher  levels  of  cognitive  control.  Interfaces  should  therefore  present 
information  that  allows  workers  to  rely  on  lower  levels  of  cognitive  control  (Naiker,  2006). 
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Naikar,  N.  (2006).  An  examination  of  the  key  concepts  of  the  five  phases  of  cognitive  work  analysis 
with  examples  from  a  familiar  system.  Proceedings  of  the  50th  Human  Factors  and  Ergonomics 
Society  Conference,  San  Francisco,  CA 


Overview: 

This  paper  examines  the  concepts  of  all  five  phases  of  CWA  with  examples  from  a  single  ‘system’, 
a  home.  A  home  is  described  as  a  highly  familiar  system  that  is  characterized  primarily  by  social  or 
intentional  constraints.  The  examples  in  this  paper  complement  the  case  study  provided  by  Vicente 
(1999)  of  DURESS,  a  thermo-hydraulic  micro  world  simulation  that  is  defined  largely  by  physical  or 
causal  constraints.  In  addition,  this  paper  examines  several  issues  relating  to  the  later  phases  of 
CWA,  including  whether  or  not  they  are  useful  or  unique,  evaluates  their  relationship  to  other 
approaches  for  work  analysis,  and  identifies  methodological  shortcomings. 


Conclusions  for  IASs: 

1 .  CWA  is  a  single  approach  to  work  analysis  that  generates  an  integrated,  multi-faceted 
description  of  a  system. 

2.  Alternative  techniques  can  be  used  in  some  or  all  of  the  phases  of  CWA  but  it  is  important  to 
ensure  that  alternative  techniques  are  consistent  with  CWA. 

3.  There  are  some  methodological  issues  with  the  concepts  of  the  phases  of  CWA.  For  instance, 
it  is  not  clear  how  to  identify  the  range  of  strategies  that  are  possible  as  opposed  to  those  that 
are  currently  being  used  by  workers.  This  problem  is  exacerbated  when  the  system  of  interest 
is  a  future,  first-of-a-kind  system. 


4.2.9  Applied  Cognitive  Work  Analysis 

Applied  Cognitive  Work  Analysis 
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076). 

Elm,  W.  (2002).  Applied  Cognitive  Work  Analysis.  ManTech  Aegis  Research  Corporation. 
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Potter,  S.,  Roth,  E.  &  Woods,  D.  (2001).  The  Development  of  a  Computer-Aided  Cognitive 
Systems  Engineering  Too[  to  Facilitate  the  Design  of  Advanced  Decision  Support  Systems.  United 


120 


States  Air  Force  Research  Laboratory:  AFRL-HE-WP-TR-2001-0125. 

Roth,  E.  (n.d.).  Trends  in  Cognitive  Analysis:  Codifying  Methods  and  Illustrating  Benefits.  CTA 
eMagazine.  Accessed  at:  http://www.ctaresource.com/eMaqazine/.  January  27th,  2007. 


Overview: 

ACWA  is  a  streamlined  version  of  CWA  that  represents  the  results  of  knowledge  elicitation  using  a 
goals-means  decomposition,  which  is  a  modified  version  of  Rasmussen’s  Functional  Hierarchy 
(Darvill  et.  al.,  2006).  The  objective  of  ACWA  is  to  facilitate  the  incorporation  of  cognitive  task 
analysis  outputs  into  the  he  design  of  decision  support  software. 

The  goal-means  decomposition  focuses  explicitly  on  the  goals  to  be  accomplished  in  the  work 
domain  (CWA),  the  relationships  between  goals,  and  the  means  to  achieve  goals,  including  the 
decisions  required,  and  the  information  required  to  make  those  decisions. 

ACWA  provides  a  practical,  step-by-step  approach  that  links  the  demands  of  the  domain  as 
revealed  by  the  cognitive  analysis  through  the  identification  of  visualizations  and  decision-aiding 
concepts  that  will  provide  effective  support. 

Process: 

ACWA  includes  the  following  steps  (Roth,  n.d.): 

•  Functional  Abstraction  Network  (FAN).  Captures  essential  domain  concepts  and 
relationships  that  define  the  problem-space; 

•  Cognitive  Work  Requirements.  Overlaid  on  the  functional  model  as  a  way  of  identifying  the 
cognitive  demands  /  tasks  /  decisions  that  arise  in  the  domain  and  require  support; 

•  Information  /  Relationship  Requirements.  Support  the  cognitive  work  identified  in  the 
“Cognitive  Work  Requirements”; 

•  Representation  Design  Requirements.  Requirements  that  define  how  the  information  / 
relationships  should  be  represented  to  practitioner(s)  to  most  effectively  support  the 
cognitive  work;  and 

•  Presentation  Design  Concepts.  Provide  physical  embodiments  of  the  representations 
specified  in  the  previous  step  (e.g.,  rapid  prototypes  that  embody  the  display  concepts). 
The  phrase  'presentation  design'  is  used  to  emphasize  that  the  resulting  'displays'  need 
not  be  visual;  they  can  be  auditory,  tactile,  or  multi-modal. 

In  the  ACWA  analysis  and  design  approach,  each  step  is  associated  with  a  design  artifact  that 
captures  the  results.  These  design  artifacts  form  a  continuous  design  thread  that  provides  a 
principled,  traceable  link  from  cognitive  analysis  to  design. 

The  figure  below  provides  a  sequence  of  analysis  and  design  steps  that  are  used  to  create  a 
continuous  design  thread  that  starts  with  a  representation  of  domain  concepts  and  relationships 
through  the  development  of  decision  support  requirements  to  creation  of  visualization  and  aiding 
concepts  and  rapid  prototypes  with  which  to  explore  the  design  concepts  (Potter  et.  al.,  2001). 
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Advantages: 

Roth  (n.d.)  outlines  the  following  strengths  for  the  ACWA  method: 

1 .  Identification  of  high-level  domain  goals  (FAN)  allows  for  development  of  novel  visualization  of 
the  non-physical  abstractions,  to  provide  more  effective  support  of  individual  and  collaborative 
decision  making  and  planning. 

2.  Organizing  operator  cognitive  requirements  around  nodes  in  the  FAN,  rather  than  organizing 
requirements  around  pre-defined  task  sequences  (as  in  traditional  approaches  to  task 
analysis)  results  in  decision-support  systems  that  have  a  decision-centered  perspective,  and 
are  thus,  able  to  support  performance  in  unanticipated  situations  as  well  as  expected 
situations. 

3.  Providing  a  step-by-step  set  of  linked  processes  from  cognitive  analysis  to  design  ensures 
traceability  of  design  elements  to  cognitive  requirements  they  are  intended  to  support. 

4.  Design  artifacts  capture  the  results  at  each  stage  of  the  process. 

5.  Application  of  this  method  leads  to  the  development  of  a  prototype. 

Disadvantages: 

1 .  Complex  method  requiring  training  and  experience. 

2.  Little  practical  difference  between  CWA  and  ACWA. 

3.  ACWA  has  few  practitioners,  and  lacks  support  tools. 

IAI  Applicability: 

1 .  Knowledge  acquisition  is  tightly  coupled  to  modeling  of  the  work  domain  as  well  as  the _ 
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development  of  Decision  Support  Systems. 

2.  The  approach  is  able  to  yield  novel  decision  support  concepts  that  were  finely  tuned  to  the 
cognitive  work  requirements  of  the  domain  (Roth,  n.d.). 

3.  Critical  decisions,  as  well  as  the  information  required  to  support  these  decisions  are  overlaid 
on  the  nodes  in  the  FAN  (Darvill  et.  al.,  2006). 

4.  The  application  of  this  method  provides  a  “decision  centred”  design  specification. 


Reference: 


Paradis,  S.,  Breton,  R.,  Elm,  W.C.,  and  Potter,  S.S.  (2002).  A  Pragmatic  Cognitive  System 
Engineering  Approach  to  Model  Dynamic  Human  Decision-Making  Activities  in  Intelligent  and 
Automated  Systems.  In  Proceedings  of  RTO  Human  Factors  and  Medicine  Panel  (HFM) 
Symposium  held  in  Warsaw,  Poland,  7-9  October  2002 


Overview: 

This  paper  briefly  overviews  the  Cognitive  System  Engineering  (CSE)  analysis  methodology.  The 
CSE  approach,  known  as  the  Applied  Cognitive  Work  Analysis  (ACWA),  is  used  to  investigate 
decision  support  R&D  efforts.  Refer  to  article  for  details  on  how  to  conduct  ACWA. 


Conclusions  for  IASs: 

1 .  Authors  report  that  CWA  is  well  suited  to  deal  with  design  issues  related  to  decision  support 
but  if  used  in  full  scale  (that  is,  all  sequential  CWA  phases  are  used)  for  a  small  problem,  it  is 
time  consuming  and  very  expensive  to  conduct. 

2.  The  ability  to  convert  requirements  into  a  sensory  presentation  (e.g.,  design  of  auditory  and 
visual  features,  etc.)  requires  considerable  skill  beyond  cognitive  work  analysis.  It  requires  an 
understanding  of  human  perception  and  its  interaction  with  the  various  presentation 
techniques.  The  authors  claim  that  the  designer  must  really  understand  what  presentation 
characteristics  implicitly  specify  the  interaction  with  the  operator’s  perception. 

3.  The  ACWA  was  found  to  be  opportunistic  and  flexible  when  new  knowledge  elicitation 
activities  arise,  and  when  the  scope  of  the  project  itself  expanded  significantly. 

4.  The  ACWA  approach  in  concert  with  a  systematic  documentation  methodology  (e.g.,  FAN) 
was  found  very  useful  in  the  AT AC  project.  The  documentation  served  as  the  main  reference 
material  to  conduct/structure  interviews  and  training  sessions  with  subject  matter  experts.  The 
documentation  was  to  also  used  during  brainstorming  design  meetings  and  was  helpful  for 
progress  review  meetings. 


4.3  Design  Methodology 

The  section  reviews  the  following  design  methodology: 

•  Joint  Application  Design  /  Development  (JAD); 

•  US  Department  of  Defence  Architectural  Framework  (DoDAF); 
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•  Explicit  Models  Design;  and, 

•  Ecological  Interface  Design. 

4.3.1  Joint  Application  Design  /  Development 


Joint  Application  Design  /  Development 


Reference: 

Cline,  A.  (n.d.).  Joint  Application  Development  (JAD)  for  Requirements  Collection  and 
Management.  White  Paper  (Carolla  Development)  http://www.carolla.com/wp-iad.htm.  Accessed 
January  28th,  2007. 

Darvill,  D.,  Kumagai,  J.  &  Youngson,  G.  (2006).  Requirements  Analysis  Methods  for  Complex 
Systems.  Technical  Report  to  Defence  Research  &  Development  Canada  Valcartier  (CR  2005- 
076). 

Klenc,  M.W.  (2001).  The  effective  methodology  for  systems  requirement  analysis. 
(http://www.umsl.edu/~sauter/analvsis/488  fOI  papers/Klenc/).  Accessed  January,  28th,  2007. 

Yatco,  M.  (1999).  Joint  Application  Design/Development. 
http://www.umsl.edu/~sauter/analvsis/JAD.html.  Accessed  January  28th,  2007. 


Overview: 

Joint  Application  Development  is  a  process  originally  developed  for  designing  a  computer-based 
system.  JAD  was  used  to  promote  the  interaction  between  IT  professionals  and  operators  to 
facilitate  an  agreement  on  requirement  and  design  specifications.  JAD  is  defined  as  a 
management  process  that  combines  operators  and  computer  specialists  to  participate  in  an 
extremely  focused  team  collaborative  workshops,  which  allows  information  systems  for  the 
operator  to  be  integrated  in  a  shorter  time  frame  (Klenc,  2001).  The  approach  was  designed  to 
increase  development  time,  and  improve  the  quality  of  the  ‘end-product’  by  incorporating  operators 
at  the  beginning  of  the  development  life-cycle. 

Process: 

The  JAD  method  centers  on  a  structured  workshop  session.  The  workshop  focuses  on  integrating 
key  operators  (stakeholders)  and  systems  professionals  together  to  resolve  issues  that  are 
inhibiting  the  design  process.  Workshops  are  effective  at  all  levels:  enterprise,  business  area, 
application,  and  project  management. 

Process  modeling  is  used  to  prepare  for  a  JAD  session.  Mock-ups  and  prototypes  can  also  be 
used  to  validate  earlier  results.  Preparation  for  a  JAD  session  may  include  elements  of  other 
methods  focusing  on  Mission,  Function,  and  Task  Analyses  to  develop  an  understanding  of  the 
topic  area.  The  JAD  process  acknowledges  and  outlines  goals,  terminology,  processes, 
requirements,  and  objectives  during  the  workshop. 

JAD  workshops  are  held  early  in  the  development  life-cycle  to  define  objectives  and  decompose 
the  domain  into  smaller  functions,  thus  defining  boundaries  and  scope.  The  goal  is  to  minimize 
documentation  and  put  critical  information  and  knowledge  in  an  explicit  format  to  be  reused  by 
other  team  members  (Darvill  et.  al.,  2006). 

Key  players  that  may  be  involved  in  the  workshop  include: 
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•  Facilitator.  Unbiased  leader  who  has  no  ties  to  the  project; 

•  Documentation  Expert.  Documents  the  decisions  and  issues; 

•  Executive  Sponsor.  Charters  the  project  (the  system  owner); 

•  Project  Manager.  Responsible  for  the  project; 

•  Business  Users.  Intended  operators  of  the  system  being  designed  (i.e.,  end-users); 

•  Systems  Experts.  Provide  inputs  in  terms  of  the  system  constraints;  and 

•  Outside  Experts.  Business  consultants  or  technology  consultants  who  provide  expertise. 

Advantages: 

1 .  Effective  technique  for  building  operator  commitment  to  the  success  of  application  systems 
through  active  participation  in  the  analysis  of  requirements  and  the  specification  of  the  system 
design. 

2.  Extensive  operator  involvement  in  systems  requirements  definition. 

3.  JAD  results  can  be  used  as  input  to  other  methods  (e.g.,  knowledge  elicitation  technique). 

4.  Workshops  facilitate  a  common  understanding  amongst  designers,  operators  and 
stakeholders. 

5.  Decisions  (and  reasoning  for  decisions)  are  well  documented. 

Disadvantages: 

1.  Extensive  preparation. 

2.  Focuses  on  system  objectives  and  process  outcomes,  as  opposed  to  the  cognitive 
components  of  the  processes. 

3.  Workshops  can  be  dominated  by  individuals. 

4.  Participants  may  be  varied  in  terms  of  their  status  within  the  company  (e.g.,  senior  managers 
versus  mid-level  employees),  impacting  the  amount  of  participation  from  individuals. 

IAI  Applicability: 

1 .  Identifies  the  system  requirements  from  an  operator  perspective. 

2.  Drives  top-priority  requirements  and  interface  concepts. 


4.3.2  US  Department  of  Defence  Architectural  Framework 

US  Department  of  Defence  Architectural  Framework 


Reference: 


Darvill,  D.,  Kumagai,  J.  &  Youngson,  G.  (2006).  Requirements  Analysis  Methods  for  Complex  Systems. 
Technical  Report  to  Defence  Research  &  Development  Canada  Valcartier  (CR  2005-076). 

US  Department  of  Defence  Architectural  Framework  Working  Group.  (2004).  DoD_Architecture  Framework 
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-  Volume  I:  Definitions  and  guidelines  (Version  1),  February  9,  2004. 

US  Department  of  Defence  Architectural  Framework  Working  Group.  (2004).  DoD  Architecture  Framework 

-  Volume  II:  Product  descriptions  (Version  1),  February  9,  2004. 

Wood,  W.G.,  Barbacci,  M.  Clements,  P.,  Palmquist,  S.,  Ang,  H.,  Bernhardt,  L.,  Dandashi,  F.,  Emery,  D., 
Sheard,  S.,  Uzzle,  L.,  Weiler,  J.  &  Krummenoehl,  A.  (2003).  DoD  Architecture  Framework  and  Software 
Architecture  Workshop  Report.  Technical  Report:  CMU/SEI-2003-TN-006. 

Wood,  W.  &  Cohen,  S.  (2003).  DoD  Experience  with  the  C4ISR  Architecture  Framework.  Architecture 
Tradeoff  Analysis  Initiative.  Technical  Note  CMU/SEI-2003-TN-027. 


Overview: 

One  of  the  key  means  for  ensuring  interoperable  and  cost-effective  military  systems  was  to  establish 
comprehensive  architectural  guidance  for  all  of  the  Department  of  Defense  (DoD).  Thus,  DoD  policy 
highlights  the  use  of  architectures  for  understanding  the  DoD  as  an  enterprise;  one  of  the  key 
developments  is  the  Department  of  Defense’s  Architecture  Framework.  DoDAF  was  developed  to  provide 
guidance  in  describing  both  war  fighting  operations  and  business  operations  and  processes. 

DoDAF  was  developed  to  ensure  that  architecture  descriptions  developed  by  the  DoD  commands, 
services,  and  agencies  are  interoperable  across  each  organization’s  operational,  systems,  and  technical 
architecture  views,  and  also  interoperate  across  joint  and  combined  organization  boundaries  (US  DoDAF 
WG,  2004). 

The  framework  provides  rules  and  guidance  for  developing  and  presenting  architecture  descriptions. 
DoDAF  defines  three  views  of  architecture  descriptions:  Operations  View  (OV),  Systems  View  (SV);  and 
Technical  Standards  View  (TV).  The  AII-DoD  Core  Architecture  Data  Model  (CADM)  defines  the  entities 
and  relationships  for  architecture  data  elements  (US  DoDAF  WG,  2004). 

Process: 

According  to  US  DoDAF  WG  (2004),  the  architecture  description  views  are  described  as  follows: 

The  OV  describes  the  tasks  and  activities,  operational  elements,  and  information  exchanges  required  to 
achieve  DoD  missions;  mission  can  include  both  war  fighting  and  business  processes.  The  OV  contains 
graphical  and  textual  products  that  identify  the  operational  nodes  and  elements,  tasks  and  activities,  and 
information  flows.  It  outlines  the  type  of  information  exchanged,  exchange  frequency,  and  information 
exchanges  that  support  tasks  and  activities. 

The  SV  is  also  a  set  of  graphical  and  textural  products  that  describe  systems  supporting  DoD  functions 
(war  fighting  and  business).  The  SV  coordinates  systems  resources  to  the  OV,  which  support  the 
operational  activities,  and  facilitate  the  exchange  of  information  among  operational  nodes. 

The  TV  is  a  rule  set  which  directs  the  arrangement,  interaction,  and  interdependencies  of  system 
elements,  ensuring  that  a  system  satisfies  operational  requirements.  The  TV  consists  of  a  collection  of 
technical  standards,  implementation  conventions,  standards  options,  rules,  and  criteria  organized  into 
profiles  that  govern  systems  and  system  elements. 

A  DoDAF  compliant  architecture  must  incorporate  explicit  linkages  among  its  various  views.  The  three 
views  and  their  interrelationships  provide  the  basis  for  measuring  system  interoperability  and  performance, 
as  well  as  their  impact  on  mission  and  task  effectiveness. 
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Relates  Systems  and  Characteristics 
to  Operational  Needs 


Operational 

View 


Identifies  What  Heeds  to  be 
Accomplished  and  Who  Does  It 


Specific  System  Capabilities 
Requ  red  Ed  Satisfy 
*  r  '“""I  E>: changes 


I ^rrmon  I 


I 


Technical  Standards 
View 


T echnical  Standards  Criteria 
Governing  Interoperable 
Implementatoni'Procyrement 
of  the  Selected  System  Capabilities 


Prescribes  Standards  and 
Conventions 


Fundamental  linkages  among  DoDAF  views  (US  DoDAF  WG  2004). 


Advantages: 

1 .  Comprehensive  architecture  that  provides  extensive  details  of  a  system’s  components. 

2.  Able  to  identify  multiple  players  within  a  system,  which  can  result  in  a  systems  of  systems  analysis. 

3.  Can  support  the  System  Engineering  approach  to  provide  a  more  rigorous  method  for  generating 
requirements. 

4.  The  information  gathered  to  develop  the  DoDAF  frameworks  can  be  used  as  valuable  data  input  for 
Human  Factors  and  Cognitive  Engineering  analysis  techniques  such  as:  Mission,  Function,  &  Task 
Analysis,  Hierarchical  Goal  Structure;  and  Cognitive  Work  Analysis. 

Disadvantages: 

1 .  Describes  what  types  of  information  need  to  be  captured  but  it  does  not  detail  how  that  information 
should  be  captured. 

2.  Although  DoDAF  documents  system  architectures,  it  does  not  address  software  architectures. 
Software  views  are  sometimes  needed  to  supplement  DoDAF  representations. 

3.  Complex  method,  involving  extensive  cost,  expertise,  and  time. 

4.  No  specific  human-related  views  within  the  framework. 

IAI  Applicability: 

1 .  Applicable  across:  concept  design,  requirements  analysis,  function  analysis,  interface  development, 
team  development,  performance,  workload,  and  training. 
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4.3.3  Explicit  Models  Design 


Explicit  Models  Design 


Reference: 

Edwards,  J.L.  (2006).  Cognitive  Style  Assessment  and  Adaptation  to  User  Style  in  the  LOCATE 
Worksapce  Layout  Design  Tool.  Defence  Research  &  Development  Canada  Toronto  (CR  2006- 
089). 

Edwards,  J.  L.  (2004).  A  Generic,  Agent  Based  Framework  for  the  Design  and  Development  of 
UAV/UCAV  Control  Systems.  Defence  Research  &  Development  Canada  Toronto,  Toronto, 
Ontairo,  Canada. 

Mason,  J.A.  &  Edwards,  J.L.  (1988).  Explicit  Models  in  Intelligent  Interface  Design.  ACM  SIGCHI 
Bulletin,  20(1). 


Overview: 

The  objective  of  Explicit  Models  Design  is  to  identify  and  define  the  knowledge  required  by 
intelligent  systems.  EMD  characterizes  and  models  knowledge  according  to  five  distinct, 
interacting  models  (Edwards,  2004),  including: 

•  Task  Model; 

•  User  Model; 

•  System  Model; 

•  Dialogue  Model;  and 

•  World  Model. 

Plan  recognition  and  plan  generation  are  also  two  processes  that  operate  within  the  EMD 
framework. 

Process: 

The  Task  Model  contains  knowledge  pertaining  to  the  tasks  an  operator  performs;  this  knowledge 
is  represented  as  a  hierarchy  of  actions,  goals  and  plans.  Satisfying  low-level  goals  allows  for  the 
attainment  and  achievement  of  higher  level  goals,  commonly  known  as  tasks.  The  pathway  from  a 
low-level  goal  to  a  high-level  goal  identifies  and  defines  a  plan  for  attaining  that  goal. 

The  System  Model  is  also  characterized  by  a  goal  hierarchy,  containing  a  description  of  the  tasks, 
goals,  and  plans  that  a  system  completes  to  support  an  operator;  these  goals  are  System  Support 
Goals.  When  multiple  system  agents  are  involved,  the  System  Model  will  be  comprised  of  a  distinct 
goal  hierarchy  for  each  agent.  The  System  Model  defines  the  level  of  assistance  provided  by  the 
system  to  aid  an  operator. 

The  User  Model  is  built  from  information  volunteered  by  an  operator,  results  of  system  requests  , 
and  from  system  monitoring  of  operator’s  activities.  The  System  Model  is  built  to  facilitate  the 
system’s  recognition  of  each  operator’s  unique  profile. 

The  Dialogue  Model  identifies  the  communication  and  interaction  that  takes  place  between  the 
operator,  system,  and  other  system  agents.  The  Dialogue  Model  must  be  built  to  allow  feedback 
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between  agents. 

The  World  Model  defines  the  external  world,  according  to  the  objects  that  exist  in  the  world,  their 
properties,  and  the  rules  that  govern  them.  These  rules  can  be  varied,  such  as  physical  rules, 
psychological  rules,  and  cultural  rules. 

Plan  Recognition  involves  the  recognition  of  operator  plans  to  enhance  the  system’s  “awareness” 
of  the  task/goal  an  operator  is  trying  to  accomplish,  and  providing  assistance  to  support  that  plan. 

Plan  Generation  pertains  to  the  system’s  ability  to  develop  and  provide  a  strategy  for  assisting  an 
operator  in  accomplishing  a  specific  task/goal.  Plan  Generation  is  based  on:  the  System  Model 
knowledge  of  a  hierarchy  of  available  support  goals  and  plans;  Task  Model  knowledge  of  an 
operator’s  current  goals  and  plans;  and  a  User  Model  knowledge  of  an  Operator’s  preferences  and 
abilities. 

Plan  recognition  and  plan  generation  can  be  associated  with  activities  in  the  Perceptual  Control 
Theory. 


Advantages: 

1 .  Supports  multi-agent  system  development. 

2.  Incorporates  the  concept  of  feedback,  defining  the  support  required  between  the  operator  and 
the  system,  allowing  one  agent  to  convey  its  goals,  plans  and  knowledge  to  another  agent. 

Disadvantages: 

1 .  Difficulty  in  characterizing  the  knowledge  according  to  one  of  the  five  models. 

2.  Difficulty  in  coordinating  the  interaction  of  knowledge  between  models. 

IAI  Applicability: 

Mason  &  Edwards  (1988)  identify  the  following  issues  pertaining  to  the  design  of  intelligent 
interfaces: 

1 .  Constraints  on  human  processing,  such  as  attention  span,  should  be  accommodated  for  in  the 
design; 

2.  An  intelligent  system  should  be  a  partly  autonomous  agent; 

3.  An  Intelligent  system  should  be  designed  to  incorporate  explicit  active  models  of  tasks, 
operators,  the  system,  and  the  dialog  with  the  operator; 

4.  An  intelligent  system  should  model  operators  in  terms  of  their  individual  characteristics;  and, 

5.  Additional  IAI  applicability  includes: 

•  Supports  multi-agent  system  development; 

•  Recognizes  Operator  and  System  roles  as  agents  and  allows  for  the  involvement  of 
both  multiple  human  operators  and  system  agents,  each  represented  by  its  own  User 
or  System  Agent  Model  (Edwards,  2004);  and, 

•  External  agents  are  recognized  as  also  possibly  supporting  the  goals  of  both  human 
operators  and  system  agents. 
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4.3.4  Ecological  Interface  Design 


Ecological  Interface  Design 


Reference: 

Edwards,  J.  L.  (2004).  A  Generic,  Agent  Based  Framework  for  the  Design  and  Development  of 
UAV/UCAV  Control  Systems.  Defence  Research  &  Development  Canada  Toronto,  Toronto, 
Ontairo,  Canada. 

Hou,  M.  (2003).  A  Framework  for  Optimizing  Operator-Agent  Interaction.  Technical  Report: 
Defence  Research  &  Development  Canada  Toronto. 

Roth,  E.M.,  Patterson,  E.S.  &  Mumaw,  R.  J.  (2001).  Cognitive  Engineering:  Issues  in  User- 
Centered  System  Design.  In  J.J.  Marciniak  (Ed.),  Encyclopedia  of  Software  Engineering,  2nd 
Edition.  New  York:  Wiley-lnterscience,  John  Wiley  &  Sons. 

Vicente,  K.J.  &  Rasmussen,  J.  (1992).  Ecological  interface  design:  theoretical  foundations.  IEEE 
TransActions  on  Systems,  Man,  and  Cybernetics,  Vol.  SMC-22,  589-606. 


Overview: 

Ecological  Interface  Design  is  an  interface  design  approach  that  incorporates  elements  from 
ecological  psychology,  particularly  with  emphasis  on  the  importance  of  considering  the  interaction 
of  humans  with  their  environment  (Vicente  &  Rasmusen,  1992).  EID  examines  how  humans 
interact  with  their  surroundings,  taking  into  account  both  physical  and  cognitive  factors,  in  the 
context  of  the  complex  system  under  control  (Edwards,  2004).  EID  provides  interface  design 
guidance,  with  the  intent  of  aiding  the  design  of  interfaces  that  are  intuitive  and  flexible  for  the 
operators,  to  ensure  optimal  usability  and  safety.  EID  also  incorporates  cognitive  factors  in 
interface  design  guidance  to  ensure  that  interface  designs  also  account  for  how  Operators  make 
decisions  and  analyse  problems. 

Process: 

Two  theoretical  concepts  underlie  the  EID  framework: 

•  Abstraction  Hierarchy.  Means-end  hierarchy  that  describes  the  properties  of  a  complex 
work  domain  (Edwards,  2004).  The  hierarchy  includes  the  following  five  levels  (based  on 
process  control): 

o  Functional  Purpose.  High-level  purpose  for  which  the  system  was  designed; 
o  Abstract  Function.  Causal  structure  of  the  system; 

o  Generalised  Function.  Decomposition  of  sub-functions  that  enable  the  high-level 
functions; 

o  Physical  Function.  Characteristics  of  system  components  and  their  connections; 
and 

o  Physical  Form.  Physical  appearance  and  location  of  components. 

System  functions  are  described  and  characterized  according  to  goals.  Moving  upwards 
through  the  levels  identifies  the  broad  goals  that  the  system  achieves.  Moving  downwards 
through  the  levels  identifies  the  method(s)  to  achieving  those  goals.  Edwards  (2004) 
_ indicates  that  this  “goal-based”  structure  has  the  potential  to  dovetail  with  the  goal  and 
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plan  hierarchy  of  Explicit  Models  Design. 

•  Skills,  Rules,  and  Knowledge  Taxonomy.  Taxonomy  of  three  principles  that  correspond  to 
three  levels  of  cognitive  control.  Vicente  &  Rasmussen  (1992)  describe  these  principles  as 
the  following: 

o  Skill-based  behaviour.  To  support  interaction  via  time-space  signals,  the  Operator 
should  be  able  to  act  directly  on  the  display,  and  the  structure  of  the  displayed 
information  should  be  isomorphic  to  the  part-whole  structure  of  movements 
(actions).  This  rule  corresponds  to  principles  of  direct  manipulation;  the  interface 
should  represent  an  Operator’s  mental  model  of  the  system,  and  the  Operator 
should  have  control  over  this  representation. 

o  Rule-based  behaviour.  Provides  a  consistent  one-to-one  mapping  between  the 
work  domain  constraints  and  the  cues  or  signs  provided  by  the  interface.  This  rule 
indicates  that  the  interface  should  provide  representations  for  all  relevant 
constraints  in  the  system. 

o  Knowledge-based  behaviour.  Represents  the  work  domain  in  the  form  of  an 
abstraction  hierarchy  to  serve  as  an  externalised  mental  model  that  will  support 
knowledge-based  problem  solving.  This  rule  indicates  that  the  interface  represent 
the  work  domain  at  all  levels  of  abstraction. 


Advantages: 

1 .  Approach  accommodates  both  perceptual  and  analytical  cognitive  processing. 

2.  Accounts  for  perceptual  and  analytical  cognitive  processing  which  facilitates  optimal  interaction 
between  system  and  operator  agents. 

3.  EID  was  developed  to  ensure  the  safety  and  reliability  of  complex  systems. 

Disadvantages: 

Edwards  (2004)  outlines  the  following  disadvantages  of  the  EID  approach: 

1 .  Does  not  address  implications  of  automation,  such  as  boredom  and  fatigue,  that  can  result 
from  repeated  applications  of  rule-based  and  skill-based  behaviours;  and 

2.  Does  not  address  how  to  detect  when  Operators  would  become  over-reliant  on  perceptual 
behaviours  and  fail  to  use  knowledge-based  skills  in  situations  where  they  are  required. 

3.  Little  experimental  research  to  support  EID  approach. 

IAI  Applicability: 

1 .  Accounting  for  perceptual  and  analytical  cognitive  processing  facilitates  optimal  interaction 
between  system  and  operator  agents,  by  accounting  for  both  skill  and  rule  based  behaviours. 

2.  EID  specifically  designs  for  complex  systems;  and 

3.  Provides  effective  mappings  between  domain  goals  and  OMI  characteristics. 
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4.4  Summary 

Tables  10,  11  and  12  summarise  the  analysis  methodologies  and  Tables  13  and  14  summarise 
the  design  methodologies.  The  methodologies  are  described  in  terms  of  their  advantages  and 
disadvantages,  and  their  applicability  to  intelligent  adaptive  systems. 
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Hierarchical  Task  Analysis 

Mission  Function  Task  Analysis 

Hierarchical  Goal  Analysis 

Advantages 

Cost  efficient  task  analysis  method. 

Easy  to  learn  and  apply. 

Results  can  provide  the  input  to  a  variety  of 
other  analyses. 

Provides  an  analytical  framework  for 
designers. 

Relatively  easy  method  for  Subject  Matter 

Experts  and  analysts. 

Able  to  use  past  experience  to  outline  the 
information  flow/activities  for  a  given  scenario. 

Task  analyses  and  data  can  be  reusable 
between  missions/systems. 

Able  to  identify  increased  workload,  and  task 
conflicts. 

PCT-based  HGA  addresses  many  of  the 
deficiencies  associated  with  traditional  HGA 
(Hendyet.  al.,  2001). 

PCT-based  HGA  provides  an  additional  step  to 
Hierarchical  Task  Analysis  such  that  it 
acknowledges  the  need  for  error  correction  at  all 
levels  within  the  hierarchy. 

HGA  can  be  integrated  into  the  engineering  design 
process. 

Considers  all  goals  (highest-level  to  lowest-level) 
as  possibilities  to  be  assigned  to  agents,  either 
human  or  machine. 

Disadvantages 

Provides  a  narrow  view  of  the  task,  and 
should  normally  be  used  in  conjunction  with 
other  task  analysis  methods  to  increase  its 
effectiveness,  and  to  develop  a  more 
complete  understanding  of  human  activity. 

Generally  used  to  describe  simple  rather  than 
complex  tasks. 

Extensive  effort  to  complete  a  MFTA. 

Individuals  conducting  the  MFTA  may  not  be  the 
same  individuals  responsible  for  designing  the 
system,  creating  the  need  for  a  large  data 
transfer. 

Considerable  time. 

Training  and  experience  required  to  implement. 

Applicability 
to  Intelligent 
Adaptive 
Systems 

Provides  a  model  for  task  execution,  enabling 
interface  designers  to  envision  the  goals, 
tasks,  subtasks,  operations,  and  plans 
essential  to  operators’  activities  (Crystal  & 
Ellington,  2004). 

Can  be  easily  extended  to  provide  system  and 
information  requirements. 

Information  needs  (both  input  and  output)  are 
typically  deduced  for  the  tasks.  These  needs, 
when  combined  with  task  relationship 
information,  can  provide  a  basis  for 
prioritizing,  clustering,  filtering,  or  sequencing 
information  presentation  in  an  interface  design 
(Miller  &  Vicente,  2001). 

MFTA  task  hierarchy  would  need  to  be  modified 
and  expanded  to  consider  intelligent  agents 
(Chow  et.  at.,  2006). 

Produces  information  and  action  requirements 
which  could  inform  training  and  interface  design, 
or  task  re-allocation  to  support  multiple 
operators/systems. 

Able  to  identify  high  areas  of  workload  and  task 
conflicts,  identifying  where  system  support  may 
be  required. 

Designing  for  control  loop  stability  between  an 
Operator  and  machine  can  ensure  that  the  proper 
controls  and  displays  are  embedded  in  an 
interface,  to  ensure  that  perceptual  errors  inherent 
in  human-machine  interaction  are  minimized. 

The  HGA  hierarchy  accounts  for  error  correction  at 
all  levels  of  the  goal  hierarchy. 

The  primary  output  from  a  HGA  analysis  is  a  goal 
structure  which  provides  interface  guidance. 

Output  from  a  HGA  analysis  will  identify  cognitive 
and  perceptual  information  related  to  Output 
Interfaces  and  Input  Interfaces. 

Tables  10, 11  and  12:  Summary  of  analysis  methodologies. 
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Goal  Directed  Task  Analysis 

Cognitive  Task  Analysis 

Team  Cognitive  Task  Analysis 

Advantages 

Details  the  SA  requirements  relevant 
to  attaining  each  goal. 

Identification  of  the  SA  requirements 
will  aid  evaluation  and  design  of 
systems  to  ensure  that  system 
supports  an  Operator  in  building  and 
maintaining  a  high-level  of  SA 

CTA  can  boost  human  performance  by  guiding  the 
development  of  tools  and  programs  that  support  the 
cognitive  processes  required  for  a  task. 

Able  to  analyze  task  performance  in  situations  that 
involve  change,  uncertainty  and  time  pressure. 

Aid  experts  in  articulating  knowledge  that  is 
generally  difficult  to  verbalize. 

The  output  to  Team  CTA  provides  input  to 
team  design,  team  performance 
measurement,  and  team  training. 

The  output  to  Team  CTA  provides  results  that 
can  act  as  input  to  other  analysis  methods. 

Useful  for  analysis  complex  multi-person 
judgements  and  decision-making. 

Aids  the  design  of  systems  and  interfaces  that 
are  used  for  teams. 

Disadvantages 

Comprehensive  method  taking 
extensive  time  to  complete; 

Requires  several  sessions  with  subject 
matter  experts  (SME)  to  define  the 
domain. 

Degree  of  subjectivity  during  the  SME 
sessions. 

CTA  encompasses  a  collection  of  diverse 
approaches  with  very  little  connection  or 
cohesiveness. 

CTA  cannot  be  viewed  as  a  standalone  analysis.  It 
needs  to  be  an  iterative  process  that  learns  from 
subsequent  design  activities. 

Extensive  time  and  expertise  required. 

Common  challenges  are  apparent  when 
conducting  group  interviews. 

Little  research  regarding  the  application  of 

Team  CTA  techniques. 

Application  of  a  method  of  analysis  designed 
for  individuals  to  teams  is  not  sufficient  for  true 
understanding  of  how  a  team  works. 

Applicability  to 
Intelligent 

Adaptive  Systems 

Identifying  what  information  an 

Operator  needs  to  know,  providing 
guidance  for  designing  a  meaningful 
interface  design. 

Identifying  functional  grouping  of 
information. 

Guiding  the  relationship  between 
information  and  decisions  to  support 
goals. 

Identifying  critical  cues  required  to 
direct  shifts  in  task  priority. 

CTA  can  boost  human  performance  by  guiding  the 
development  of  tools  and  programs  that  support  the 
cognitive  processes  required  for  a  task. 

CTA  must  work  within  a  system  development 
process  and  support  critical  system  design  issues. 

Views  a  team  as  an  intelligent  entity,  and 
attempt  to  identify  the  cognitive  processes 
required  by  team  dependent  tasks. 

Captures  the  cognitive  processes  of  a  team, 
and  focuses  on  the  way  a  team  coordinates 
the  understanding  of  the  different  members 
and  synthesizes  the  task  elements. 

Used  to  identify  information,  cues,  and 
strategies  required  to  make  key  decisions. 
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Applied  Cognitive  Task 
Analysis 

Cognitive  Work  Analysis 

Applied  Cognitive  Work  Analysis 

Advantages 

ACTA  techniques  are  easy  to  use, 
flexible,  and  provide  clear  output. 

Identifies  where  a  system’s  design 
must  support  human  problem-solving 
and  decision-making  by  assessing 
complex  tasks  that  require  a  high 
degree  of  cognitive  skill. 

Results  from  CWA  can  be  transferred  directly 
to  design  requirements. 

Accounts  for  the  role  of  the  workers  in  complex 
systems. 

Also  focuses  on  analysing  the  environment. 

The  model  provides  traceability  of  decision 
making  in  a  organisational  structure. 

The  model  provides  a  link  between  the  abstract 
functions  in  the  higher  hierarchy  level  and  the 
plans/courses  of  action  in  the  lower  hierarchy 
level. 

Identification  of  high-level  domain  goals  (FAN)  allows 
for  development  of  novel  visualization  of  the  non¬ 
physical  abstractions,  provide  more  effective  support 
of  individual  and  collaborative  decision  making  and 
planning. 

Organizing  operator  cognitive  requirements  around 
nodes  in  FAN,  rather  than  organizing  requirements 
around  predefined  task  sequences  (as  in  traditional 
approaches  to  task  analysis),  results  in  decision- 
support  systems  that  have  a  decision-centered 
perspective,  and  are  thus  able  to  support 
performance  in  unanticipated  situations  as  well  as 
expected  situations. 

Providing  a  step-by-step  set  of  linked  processes  from 
cognitive  analysis  to  design  insures  traceability  of 
design  elements  to  cognitive  requirements  they  are 
intended  to  support. 

Design  artefacts  capture  the  results  at  each  stage  of 
the  process. 

Application  of  this  method  leads  to  the  development 
of  a  prototype. 

Disadvantages 

Although  ACTA  elicits  important 
cognitive  information,  there  is  a  trade¬ 
off  when  using  a  streamlined 
approach;  the  more  streamlined  and 
proceduralized  CTA  techniques 
become,  the  less  powerful  they  are 
(Militello  &  Hutton,  1998). 

ACTA  techniques  may  gather  less 
comprehensive  information  than  more 
systematic  techniques. 

Complex  method  requiring  considerable 
expertise. 

Extensive  time  required  to  learn  and  use. 

Difficult  to  define  and  map  the  system  on  all 
five  stages. 

Little  practical  difference  between  CWA  and 
ACWA.  Many  times,  CWA  will  only  be  applied 
using  the  first  few  stages,  which  resembles 
more  of  the  ACWA  process. 

CWA  is  more  of  an  academic  endeavour  with 
more  attention  being  placed  on  completing  the 
process  as  opposed  to  using  the  analysis  to 

Complex  method  requiring  training  and  experience 

Little  practical  difference  between  CWA  and  ACWA. 

ACWA  has  few  practitioners,  and  lacks  support  tools. 
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drive  design  and  develop  design  concepts. 

Applicability  to 
Intelligent  Adaptive 
Systems 

ACTA  provides  data  that  translates 
more  directly  into  applied  products 
such  as  improved  training  scenarios  or 
interface  recommendations. 

Allows  systems  designers  to  elicit  and 
represent  critical  cognitive 
components  of  skilled  task 
performance,  and  the  means  to 
transform  these  data  into  design 
recommendations. 

ACTA  techniques  were  developed  to 
elicit  critical  cognitive  task  components 
from  Subject  Matter  Experts. 

Identifies  the  constraints  on  information 
seeking,  including  the  individual  resources  and 
the  external  environment. 

CWA  investigates  the  information  behaviour  in 
context,  therefore  the  results  are  valid  for  the 
design  of  information  systems  in  the  context 
investigated,  rather  then  for  the  design  of 
general  information  systems  (Fidel  &  Pejtersen 
(n.d.). 

The  framework  facilitates  an  in-depth 
examination  of  the  various  dimensions  of  a 
context.  A  study  of  a  particular  context  is, 
therefore,  a  multi-disciplinary  examination  with 
the  purpose  of  understanding  the  interaction 
between  people  and  information  in  the  work 
context  (Fidel,  R.  &  Pejtersen,  A.  (n.d.). 

Provides  a  structure  of  human-information 
interaction  analysis,  rather  than  subscribing  to 
specific  theories  or  models  (Fidel  &  Pejtersen 
(n.d.). 

Workers  will  implement  lower  levels  of  cognitive 
control  more  quickly,  effectively,  and 
effortlessly  than  higher  levels  of  cognitive 
control.  Interfaces  should  therefore  present 
information  that  allows  workers  to  rely  on  lower 
levels  of  cognitive  control  (Naiker,  2006) 

Knowledge  acquisition  is  tightly  coupled  to  modeling 
of  the  work  domain  as  well  as  the  development  of 
Decision  Support  Systems.’ 

The  approach  is  able  to  yield  novel  decision  support 
concepts  that  were  finely  tuned  to  the  cognitive  work 
requirements  of  the  domain. 

Critical  decisions  as  well  as  the  information  required 
to  support  the  decisions  are  overlaid  on  the  nodes  in 
the  FAN. 

The  application  of  this  method  provides  a  “decision 
centred”  design  specification. 
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Tables  13  and  14:  Summary  of  design  methodologies. 


Joint  Application  Design  /  Development 

US  Department  of  Defence  Architectural 
Framework 

Advantages 

Effective  technique  for  building  operator  commitment  to  the 
success  of  application  systems  through  active  participation  in  the 
analysis  of  requirements  and  the  specification  of  the  system 
design. 

Extensive  operator  involvement  in  systems  requirements 
definition. 

JAD  results  can  be  used  as  input  to  other  methods  (e.g., 
knowledge  elicitation  technique). 

Workshops  facilitate  a  common  understanding  amongst 
designers,  operators  and  stakeholders. 

Decisions  (and  reasoning  for  decisions)  are  well  documented. 

Comprehensive  architecture  that  provides  extensive  details  of  a 
system’s  components. 

Able  to  identify  multiple  players  within  a  system,  which  can  result  in  a 
systems  of  systems  analysis. 

Can  support  the  System  Engineering  approach  to  provide  a  more 
rigorous  method  for  generating  requirements. 

The  information  gathered  to  develop  the  DoDAF  frameworks  can  be 
used  as  valuable  data  input  for  Human  Factors  and  Cognitive 
Engineering  analysis  techniques  such  as:  Mission,  Function,  &  Task 
Analysis,  Hierarchical  Goal  Structure;  and  Cognitive  Work  Analysis. 

Disadvantages 

Extensive  preparation. 

Focuses  on  system  objectives  and  process  outcomes,  as 
opposed  to  the  cognitive  components  of  the  processes. 

Workshops  can  be  dominated  by  individuals. 

Participants  may  be  varied  in  terms  of  their  status  within  the 
company  (e.g.,  senior  managers  versus  mid-level  employees), 
impacting  the  amount  of  participation  from  individuals. 

Describes  what  types  of  information  need  to  be  captured  but  it  does 
not  detail  how  that  information  should  be  captured. 

Although  DoDAF  documents  system  architectures,  it  does  not 
address  software  architectures.  Software  views  are  sometimes 
needed  to  supplement  DoDAF  representations. 

Complex  method,  involving  extensive  cost,  expertise,  and  time. 

No  specific  human-related  views  within  the  framework 

Applicability  to 
Intelligent  Adaptive 
Systems 

Identifies  the  system  requirements  from  an  operator  perspective. 

Drives  top-priority  requirements  and  interface  concepts. 

Applicable  across:  concept  design,  requirements  analysis,  function 
analysis,  interface  development,  team  development,  performance, 
workload,  and  training 
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Explicit  Models  Design 

Ecological  Interface  Design 

Advantages 

Supports  multi-agent  system  development. 

Incorporates  the  concept  of  feedback,  defining  the  support 
required  between  the  operator  and  the  system,  allowing  one 
agent  to  convey  its  goals,  plans  and  knowledge  to  another 
agent. 

Approach  accommodates  both  perceptual  and  analytical  cognitive 
processing. 

Accounting  for  perceptual  and  analytical  cognitive  processing  facilitates 
optimal  interaction  between  system  and  operator  agents. 

EID  was  developed  to  ensure  the  safety  and  reliability  of  complex  systems. 

Disadvantages 

Difficulty  in  characterizing  the  knowledge  according  to  one  of 
the  five  models. 

Difficulty  in  coordinating  the  interaction  of  knowledge  between 
models 

Does  not  address  implications  of  automation,  such  as  boredom  and  fatigue, 
that  can  result  from  repeated  applications  of  rule-based  and  skill-based 
behaviours;  and 

Does  not  address  how  to  detect  when  Operators  would  become  over-reliant 
on  perceptual  behaviours  and  fail  to  use  knowledge-based  skills  in 
situations  where  they  are  required. 

Little  experimental  research  to  support  EID  approach. 

Applicability  to 
Intelligent  Adaptive 
Systems 

Constraints  on  human  processing,  such  as  attention  span, 
should  be  accommodated  for  in  the  design; 

An  intelligent  system  should  be  a  partly  autonomous  agent; 

An  Intelligent  system  should  be  designed  to  incorporate  explicit 
active  models  of  tasks,  operators,  the  system,  and  the  dialog 
with  the  operator;  and 

An  intelligent  system  should  model  operators  in  terms  of  their 
individual  characteristics. 

Supports  multi-agent  system  development;  and 

Recognizes  Operator  and  System  roles  as  agents  and  allows 
for  the  involvement  of  both  multiple  human  operators  and 
system  agents,  each  represented  by  its  own  Operator  or 

System  Agent  Model  (Edwards,  2004). 

External  agents  are  recognized  as  also  possibly  supporting  the 
goals  of  both  human  operators  and  system  agents 

Accounting  for  perceptual  and  analytical  cognitive  processing  facilitates 
optimal  interaction  between  system  and  operator  agents,  by  accounting  for 
both  skill  and  rule  based  behaviours. 

EID  specifically  designed  for  complex  systems;  and 

Provides  effective  mappings  between  domain  goals  and  OMI 
characteristics. 
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5  Agent-Based  Design  Principles 


5.1  Introduction 

Intelligent  Adaptive  Systems  can  be  considered  in  the  context  of  a  human-electronic 
crewmember  team  involving  collaborative  and  co-operative  interaction  between  the  human 
and  the  computer-based  agents  (see  Taylor,  1997).  As  such,  research  relating  to  understanding 
and  aiding  human  interaction  in  real-world  systems  is  critical.  This  review  will  examine  issues 
relating  to  this  collaborative  co-operative  environment,  such  as  human-agent  teamwork, 
organisation,  and  interaction. 

Edwards  (2004)  defines  autonomous  software  agents  as  programmes  that  have  the  ability  to 
sense  their  environment  and  act  on  the  environment  over  time  to  achieve  a  particular  goal. 
Agents  can  be  communicative  (i.e.,  interact  with  other  agents  or  people),  adaptive  /  learning 
(i.e.,  can  change  their  behaviour  based  on  past  experience  of  interacting  with  the  operator), 
and  mobile  (i.e.,  can  move  themselves  from  one  machine  to  another).  Edwards  argues  that 
agents  offer  number  advantages  for  the  development  of  IASs;  particularly,  agents  acting 
intelligently  and  autonomously  are  ideally  suited  to  attaining  control  of  some  tasks  from  the 
human  operators.  Edwards  also  describes  how  the  development  of  agents  can  be  integrated 
into  the  CommonKADS  and  Explicit  Models  Design  approaches  (Section  9.3.3). 

5.1.1  Background 

A  prominent  factor  limiting  tactical  performance  is  the  inability  of  operators  to  realise  the  full 
potential  of  their  equipment.  One  aspect  of  this  problem  is  that  operators  cannot  process  all 
the  information  presented  to  them  in  the  limited  time  available.  Advances  in  automation  have 
aided  the  operator  in  this  task.  Though  conventional  (i.e.  fixed  or  static)  automation  can 
reduce  pilot  workload,  automated  systems  have  forced  pilots  to  act  in  an  increasingly 
supervisory  capacity.  This  has  led  to  an  increase  in  errors  associated  with  monitoring  control 
by  the  operator  with  a  consequent  impact  on  task  performance.  A  myth  about  the  impact  of 
automation  on  human  performance  is  that,  as  investment  increases,  less  investment  is  needed 
in  human  expertise.  In  fact,  increased  automation  creates  new  knowledge  and  skill 
requirements.  Today,  this  issue  is  even  more  relevant  due  to  more  capable  technology  and  the 
increased  potential  for  automation  and  support  at  higher  levels.  Banbury  (1997)  summarises  a 
number  of  problems  associated  with  increased  automation: 

•  Increased  monitoring  load.  The  automation  of  functions  will  leave  the  operator  with 
fewer  functions  to  execute,  but  with  a  more  complex  system  to  monitor;  a  function  at 
which  humans  do  not  excel; 

•  “Out-of-the-loop” performance  problems.  Numerous  studies  (see  section  10.1.1  for  a 
review)  have  shown  that  the  implementation  of  automation  may  make  humans  slower 
and  less  accurate  at  failure  detection  when  they  become  passive  as  compared  to  active 
decision  makers.  Situation  Awareness  is  one  of  the  primary  factors  underlying  out-of- 
the-loop  performance  problems;  SA  suffers  as  the  operator  becomes  a  passive 
decision  maker; 
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•  Loss  of  skills.  In  relation  to  the  out-of-the-loop  problem,  a  loss  of  skills  may  also 
result,  rendering  operators  less  able  to  perform  functions  when  they  resume  manual 
control  following  an  automation  failure; 

•  Over-trust  (i.e.  complacency)  and  under-trust  (i.e.,  scepticism) .  Operators  may 
possess  either  too  much  trust  in  automated  systems,  leading  to  a  false  sense  of 
complacency  and  lack  of  proper  monitoring,  or  a  complete  lack  of  trust,  characterised 
by  complete  disuse  of  the  system,  even  when  it  might  be  beneficial.  Both  result  in 
sub-optimal  performance,  and  the  latter  also  creates  an  increase  in  workload. 

•  Increased  system  complexity.  The  addition  of  automation  tends  to  increase  system 
complexity;  not  only  is  the  initial  system  present,  but  the  new  system  then  automates 
a  function,  which  means  more  components  to  monitor  and  more  systems  for  the 
operator  to  understand.  Furthermore,  there  is  an  increased  probability  of  system 
failure  associated  with  the  increased  number  of  systems,  adding  to  the  complexity  of 
the  operator’s  role. 

The  potential  for  automation-induced  error  has  raised  concerns  over  possible  losses  in 
operator  SA  and  experiencing  difficulty  in  returning  to  active  control  when  necessary. 

Reducing  pilot  workload,  while  maintaining  SA,  can  only  be  accomplished  by  adopting  a 
human-centred,  as  opposed  to  technology-centred,  approach  to  cockpit  automation.  Operators 
should,  therefore,  play  a  more  active  role  in  the  control  loop;  the  reductions  in  operator 
workload  are  manifest  from  the  system  improving,  rather  than  replacing,  the  operator’s 
decision  making  ability. 

Wm 

Parasuraman,  R.  and  Riley,  V.  (1997).  Humans  and  automation:  Use,  misuse,  disuse  and  abuse. 
Human  Factors,  39(2),  230-253. 


Overview: 


Various  theories  and  empirical  studies  pertaining  to  human  use,  misuse,  disuse  and  abuse  of 
automation  are  reviewed  in  this  paper.  The  results  of  this  review  led  to  recommendations  for  the 
improvement  of  system  design  and  training  methods  and  policies  and  procedures  involving 
automation  use. 

The  author  defines  automation  and  the  role  that  humans  play  in  automated  systems.  A  review  of 
incidents  and  accidents  with  automated  systems  shows  that  there  are  problems  with  automation 
related  to  automation  usage  decisions  and  a  misuse  of  when  and  why  to  use  it. 

The  authors  define  the  terms  misuse,  disuses  and  abuse  of  automation: 

•  Misuse:  an  over-reliance  on  automation; 

•  Disuse:  underutilization  of  automation;  and, 

•  Abuse  inappropriate  application  of  automation. 

These  issues  can  be  influenced  by  extraneous  factors,  which  in  turn  influence  operator 
performance.  The  factors  that  can  influence  the  use  of  automation  include: 

•  Individual  attitudes  towards  automation; 

•  While  unclear,  there  is  some  evidence  that  task  load  can  influence  automation; 

•  Cognitive  overhead  can  influence  operators  not  to  use  automation  if  the  operator  does  not 
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believe  that  its  initiation  can  overcome  the  cognitive  overhead  (takes  more  work  to  implement 
than  if  they  did  it  themselves); 

•  Reliability  and  operator  trust;  and, 

•  Self-confidence,  risk,  individual  differences  can  also  influence  automation  use  either  directly 
or  indirectly  as  a  moderator. 

The  authors  outline  factors  that  can  influence  the  misuse  of  automation: 

•  Over-reliance  on  automation  can  lead  to  monitoring  errors; 

•  Operator  decision  biases  such  as  using  decision  heuristics  may  lead  to  monitoring  failures, 
commission  errors,  and  over-reliance  on  automation;  and, 

•  Automation  reliability  and  consistency  can  lead  to  over-reliance. 

The  authors  outline  some  factors  that  may  influence  the  disuse  of  automation: 

•  Mistrust  of  the  automation  may  lead  operators  to  disable  or  ignore  (e.g.,  alerts)  automation. 

The  authors  outline  some  factors  that  may  influence  the  abuse  of  automation: 

•  Management  practices  or  corporate  policies  may  prevent  the  use  of  automation; 

•  Indiscriminate  application  of  automation  by  using  a  technology-centered  approach  without 
considering  the  resulting  roles  and  responsibilities  of  the  operator;  and, 

•  The  automation  is  granted  too  high  a  level  of  authority  without  appropriate  feedback,  and 
removing  the  operator  from  the  decision  making  process  of  when  and  how  to  use  the 
automation  may  lead  to  complacency,  decreased  trust,  and  monitoring  performance. 


Conclusions  for  IASs: 

•  Results  from  reviewing  the  factors  influencing  automation  use  suggest  that  individual 
operators  should  be  made  aware  of  their  biases  (due  to  individual  differences)  on  automation 
use.  Other  recommendations  are: 

o  Better  operator  knowledge  of  how  automation  works; 

o  Implementation  of  policies  and  procedures  that  highlight  the  importance  of  when 
and  where  to  use  automation; 

o  Teach  operators  to  make  rational  automation  use  decisions;  and, 

o  Make  automation  easy  and  efficient  to  use. 

•  Results  from  reviewing  the  factors  influencing  automation  misuse  suggests  that  making 
automation  state  indicators  and  adaptive  task  allocation  to  enhance  operator  involvement, 
and  using  display  techniques  may  enhance  operator  monitoring  performance.  Other 
recommendations  are: 

o  System  designers,  regulators  and  operators  should  be  taught  to  recognize 
automation  over-reliance  and  its  consequences; 

o  Let  operators  use  automation  cues  as  heuristics  for  making  decisions;  and, 

o  Provide  appropriate  and  accurate  feedback  of  automation  state. 

•  Results  from  reviewing  factors  influencing  automation  disuse  suggests  that  designers  of 
alerting  systems  must  account  for  the  decision  threshold  (false  alarms  versus  sensitive 
alarms),  and  the  base  rate  of  the  dangerous  condition  to  enhance  operator  trust  and  use  of 
automation. 

•  Designers  of  alerting  systems  should  consider  using  alarms  that  show  the  “likelihood”  of  a 

dangerous  situation  rather  than  having  the  operator  rely  on  the  automation  as  the  final _ 
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authority  (essentially  putting  the  human  in  the  loop  for  decision  making). 

•  Results  from  reviewing  the  factors  influencing  automation  abuse  suggests  the  following 
recommendations: 

o  Define  an  operator’s  role  based  on  the  operator’s  responsibilities  and  capabilities, 
and  not  because  the  technology  is  simply  available. 

o  Design  the  system  to  encourage  active  operator  involvement. 


Reference: 


Manzey,  D.,  Bahner,  J.E.,  and  Hueper,  A.D.  (2006).  Misuse  of  automated  aids  in  process  control: 
complacency,  automation  bias  and  possible  training  interventions.  Proceedings  of  the  50th  Human 
Factors  and  Ergonomics  Society 


Overview: 

The  paper  outlines  a  study  to  investigate  complacency  effects  when  operators  interact  with  an 
automated  aid  in  a  process  control  simulation  task.  Possible  performance  consequences  (i.e. 
automation  bias  in  terms  of  commission  errors,  and  impairments  of  return-to-manual-performance 
in  case  of  automation  breakdown)  are  also  examined.  The  effect  of  a  specific  training  intervention 
to  reduce  complacency  by  exposing  participants  intentionally  to  automation  failures  is  also 
investigated. 

The  results  provide  clear  evidence  for  complacency  effects  due  to  insufficient  verification  of 
recommendations  provided  by  the  automated  aid. 


Conclusions  for  IASs: 

1 .  The  authors  suggest  that  confronting  operators  with  rare  automation  failures  during  training 
may  be  a  suitable  way  to  reduce  complacency,  yet  may  not  be  sufficient  to  prevent 
complacency  effects  completely. 

2.  The  risk  of  commission  error  was  associated  with  comparatively  high  levels  of  complacency 
only;  usually  less  information  needs  to  be  sampled  to  falsify  an  automatically  generated 
diagnosis  than  to  verify  it  completely. 


5.2  General  Design  Principles 

A  number  of  researchers  have  developed  sets  of  principles  of  adaptive  systems  design.  These 
principles  can  be  classified  into  those  concerned  with  adaptation  and  those  with  interaction. 
Both  sets  of  principles  are  summarised  as  follows: 

5.2.1  Principles  of  Adaptation 

•  The  requirement  for  aiding  is  not  only  dependent  on  impending  tasks,  but  is  also 
contingent  on  recently  completed  tasks  (Morris  and  Rouse,  1986); 
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•  If  operator  modelling  is  used  to  determine  the  intervention  of  the  aid,  the  success  of 
this  approach  will  depend  upon  the  amount  of  structure  in  the  task.  Tasks  that  require 
high  levels  of  judgement  by  the  operator  may  not  be  suitable  candidates  for  the 
application  of  this  approach  (Rouse,  Geddes  and  Curry,  1987); 

•  If  a  model  can  provide  perfect  predictions  of  an  operator’s  intentions  and  actions, 
there  is  no  need  to  communicate  adaptation  explicitly,  and  thus  the  cost  of  explicit 
communication  can  be  avoided.  However,  as  uncertainty  increases,  predictions  will 
frequently  be  wrong,  and  as  a  result,  tasks  will  “slip  through  the  cracks”  or  receive 
redundant  efforts.  To  avoid  these  possibilities,  increased  explicit  communication  is 
required  to  check  or  calibrate  a  model’s  prediction  (Morris  and  Rouse,  1986);  and, 

•  If  possible,  the  IAS  should  have  the  capability  to  predict  the  effect  of  individual 
differences  on  the  efficiency  of  how  cockpit  tasks  are  conducted.  Such  assessments 
could  provide  assistance  in  deciding  the  most  appropriate  level  of  aiding  that  an 
operator  may  require  in  a  given  situation  (Morris  and  Rouse,  1986;  Lehner,  Cohen, 
Thompson,  and  Laskey,  1987). 


Reference: 


Miller,  C.A.  and  Dorneich,  M.C.  (2006).  From  Associate  Systems  to  Augmented  Cognition  25 
Years  of  User  Adaptation  in  High  Criticality  Systems.  Poster  presented  at  the  Augmented 
Cognition  conference,  October  2006,  San  Francisco. 


Overview: 

In  the  1980’s,  the  U.S.  Air  Force  initiated  the  development  of  a 
human-adaptive,  information,  and  automation  management 
technology  known  as  the  “Pilot’s  Associate”. 

What  is  it?  PA,  and  all  of  the  subsequent  associate  systems, 
consisted  of  an  integrated  suite  of  intelligent  subsystems  that 
were  designed  to  share  (among  themselves  and  with  the  pilot) 
a  common  understanding  of  the  mission,  the  current  state  of 
the  world,  the  aircraft  and  the  pilot.  Associate  systems  were 
designed  to  use  the  shared  knowledge  to  plan  and  suggest  courses  of  action,  and  to  adapt  cockpit 
information  displays  and  the  behaviour  of  aircraft  automation. 

Automation  of  tasks:  Tasks  are  automated  only  in  line  with  the  operator’s  goals  and,  whenever 
feasible,  to  be  authorized  by  the  operator.  Operator  control  of  the  automation  was  established 
either  during  the  mission,  immediately  prior  to  execution  of  automation,  or  pre-mission,  in  a  pre¬ 
authorization  mode. 

Lessons  Learned  from  PA  efforts:  Associate  systems  were  and  are  the  predecessors  of 
augmented  cognition  (AugCog)  technologies.  While  there  are  many  similarities  between  PA  and 
AugCog  systems,  there  are  also  some  many  differences: 

•  Associate  systems  leave  the  pilot  “in  charge”  which  is  extremely  important  in  high  criticality 
domains.  To  increase  the  chance  of  operator  acceptance,  it  is  important  to  consider  that  the 
operator  should  be  kept  in  the  loop.  The  authors  claim  that  if  the  pilot  is  responsible  for  the 
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actions  of  the  aircraft,  then  the  pilot  must  be  the  final  authority  of  the  aircraft’s  actions. 

•  All  components  (e.g.,  sensors,  information  fusion  technologies,  and  interfaces)  should  be  co¬ 
developed  and  evaluated  in  concert. 

•  A  task-based  framework  was  an  effective  way  to  coordinate  a  variety  of  processes  (or 
subsystems)  and  minimize  the  costs  of  revising  or  extending  them. 

•  Personification  (or  customization)  is  out  of  place  in  high  criticality  domains  (and  possibly  other 
domains,  as  HCI). 

•  OMI  design  proved  to  be  an  important  component  of  the  associate  system.  It  was  often  found 
that  the  OMI  will  highlight  anything  that  is  wrong  with  any  module,  and  errors  in  the  design  of 
the  OMI  will  make  all  other  aspects  of  the  associate  less  effective. 

•  Adaptation  of  systems  to  individual  differences  and  operator  expectations  (but  not 
customization)  can  have  large  payoffs  for  fitting  a  system  to  an  operator’s  needs  and 
capabilities. 


Conclusions  for  IASs: 

Several  “Lessons  Learned”  from  the  PA  efforts  were  outlined  in  this  paper,  which  have  implications 
for  the  development  of  IA  systems  (see  Lesson  learned  from  PA  efforts  above): 

1 .  Importance  of  operator  acceptance  and,  therefore,  importance  of  keeping  human  “in 
charge”.  The  authors  advocate  migrating  control  to  a  supervisory  level  (where  the  human 
varies  the  amount  and  level  of  automation)  and  that  the  system  should  not  rely  too  heavily 
on  inferred  operator  state  or  intent.  This  can  increase  human  out-of-the-loop  problems. 

2.  Importance  of  co-development  and  progressive  testing :  Development  efforts  and  individual 
technologies  should  be  co-developed  and  used  in  collaboration,  which  can  aid  the 
development  of  an  overall  system.  For  instance,  the  development  of  neurophysiological 
sensors  or  “meters”,  other  means  of  assessing  operator  state,  and  the  development  of 
methods  for  “augmenting”  cognition  through  information  display  technologies  need  to  be 
co-developed  and  evaluated  in  concert. 

3.  Benefits  of  an  explicit,  integrative  framework  (task  model).  Knowledge  of  the  task  context 
can  help  develop  systems  that  manage  task  demand  and  increase  operator  performance. 

4.  Operator-machine  interactions.  More  effective  means  of  interactions  between  the  operator 
and  the  system  may  be  achieved  if  the  designer  approaches  an  intelligent  system  as  a 
“personified  agent”  whose  goal  it  is  to  aid  the  operator  and  to  recognize  that  the  operator 
might  have  feelings  or  attitudes. 

5.  Importance  of  interface  and  interaction  design.  A  system,  especially  for  a  high  criticality 
domain,  should  be  designed  with  a  system  failure  in  mind.  The  authors  provide  some 
methods  that  can  accomplish  this:  give  the  operator  the  ability  to  override  and  turn  off  the 
technology;  allow  the  operator  to  explicitly  authorize  a  display  modification,  to  be  notified  of 
pending  changes,  to  be  notified  of  executed  changes,  and  to  rapidly  return  to  a  previous 
display  state. 

6.  Importance  of  learning,  especially  individuation.  Recording  individual  performance  effects 
could  serve  to  provide  a  powerful  means  of  adapting  system  behaviour  to  the  individual. 
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Albery,  W.,  and  Khomenko,  M.N.  (2002).  Differences  in  Pilot  Automation  Philosophies  in  the  US 
and  Russian  Air  Forces  Ground  Collision  Avoidance  Systems.  In  Proceedings  of  RTO  Human 
Factors  and  Medicine  Panel  (HFM)  Symposium  held  in  Warsaw,  Poland,  7-9  October  2002. 


Overview: 

This  paper  details  two  automated  collision  avoidance  systems  designed  by  the  US  and  by  Russia, 
and  distinguishes  between  the  roles  of  the  human  in  both  systems. 

The  US  Air  Force  developed  a  Ground  Collision  Avoidance  System  (GCAS)  that  is  automatic  and 
requires  no  pilot  intervention.  The  underlying  philosophy  of  this  system  is  reliability,  pilot 
unobtrusiveness,  and  invisibility.  Russia  also  developed  a  pilot  state  monitoring  system  that  is 
automatic,  but  includes  the  pilot  in  its  control  loop  (IKSL).  The  Russian  system  includes  an 
onboard  video  camera  that  allows  ground  operators  to  observe  the  pilot  during  the  mission. 

The  Auto  GCAS  solved  the  problem  of  a  disabled  or  disoriented  pilot  by  providing  a  “safety  net”  of 
a  minimum  altitude  the  aircraft  can  penetrate.  The  IKSL  relies  on  the  judgment  of  the  ground 
controllers  to  interpret  the  signals  from  the  on-board  system  and  to  “take  over”  the  aircraft,  if 
required.  The  Auto  GCAS  does  not  rely  on  any  inputs  from  the  pilot;  it  is  aircraft  state  dependent. 
The  IKSL  has  the  pilot  in  the  loop,  and  must  have  signals  from  the  pilot  in  order  to  operate. 

The  difference  in  philosophies  between  the  two  systems  reflects  the  philosophies  of  the  air  forces 
of  the  two  countries.  The  authors  suggest  that  the  American  fighter  pilots  would  probably  not  turn 
on  the  GCAS  during  actual  combat  missions,  that  they  are  intolerant  of  false  positives,  and  are 
uneasy  about  relinquishing  complete  control  of  their  aircraft  to  the  system.  The  Russian  approach 
meanwhile,  is  in  the  creation  of  a  “partner  system”  that  can  help  and  assure  the  pilot  in  dangerous 
and  emergency  situations. 


Conclusions  for  IASs: 

1 .  The  authors  conclude  that  implementing  automation  must  consider  cultural  aspects. 

2.  It  is  recommended  that  fully  automated  systems  (at  all  roles/stage  of  information  processing) 
should  not  be  used  in  complex  environments,  particularly  when  serious  consequences  may 
result  from  system  failure. 

3.  Shared  agent  and  authority  of  roles  can  be  an  optimal  approach  to  help  and  assure  operators 
in  dangerous  and  emergency  situations  (Russian  model). 
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Overview: 

This  paper  discusses  the  applicability  of  adaptability  and  adaptivity  features  to  learning  systems. 
The  paper  also  discusses  the  adaptation  needs  of  learning  systems,  with  particular  focus  on 
Intelligent  Learning  Systems  (ILS).  A  comparative  study  of  office  application  systems  was 
completed  which  have  been  an  important  research  area  in  the  field  of  adaptation  facilitation. 

Within  the  human-computer  interaction  literature,  a  model  for  levels  of  adaptivity  has  been 
proposed  by  Oppermann  (1994;  1997).  Adaptivity  can  be  thought  of  the  full  range  of  a  continuous 
spectrum,  representing  the  degree  of  operators’  control  and  involvement.  Oppermann’s  spectrum 
of  adaptivity  is  anchored  on  one  end  by  fully  adaptive  interfaces,  representing  no  operator 
involvement  or  control,  and  on  the  other  end  by  fully  adaptable  interfaces,  representing  full 
operator  involvement  and  control. 
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Conclusions  for  IASs: 

1 .  To  provide  adequate  knowledge  acquisition,  the  system  requires  evaluation  of  the  operator's 
behaviour  without  permitting  the  operator  to  modify  the  system's  assumptions  about  his/her 
behaviour. 
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Overview: 

Provides  a  good  review  of  the  benefits  and  costs  of  automatic  and  operator-controlled  systems, 
and  analyzes  how  they  have  and  have  not  resolved  problems  of  contemporary  interactive  systems. 
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To  further  understand  this  problem  from  a  more  global  perspective,  additional  models  from  HCI, 

HF  and  automation  domains  are  reviewed. 

•  Costs  associated  with  automatic  systems  include  the  potential  to  increase  workload  and  out-of- 
the-loop  performance  problems. 

•  Adaptable  OMIs  seem  to  relieve  the  operator  from  feeling  “out  of  control”  is  however  they  can 
result  in  an  increase  workload  and  a  loss  in  performance,  since  time  taken  away  from 
completing  tasks  and  directed  towards  customizing  the  interface. 

•  While  mixed-initiative  systems  afford  operators  more  control  over  automation  and  its  initiation, 
operators  still  report  feeling  out  of  control.  In  addition,  mixed-initiative  interfaces  typically 
require  operators  to  stop  work  and  accept  or  reject  recommendations,  with  some  interfaces 
requiring  operators  to  then  implement  the  change.  This  may  potentially  hinder  operators  who 
already  have  a  large  workload  and  are  working  in  a  complex  dynamic  environment. 

Roberts  (2006),  in  a  comprehensive  review  of  adaptive  interfaces  and  automation,  has  proposed  a 
framework  that  aims  to  outline  the  problem  space  of  adaptive  interfaces  and  automation,  and  find  a 
balance  between  workload  and  operator  control. 


Conclusions  for  IASs: 

1 .  The  amount  of  operator  control  and  involvement  (adaptivity  and  automation)  and  the  roles  that 
humans  and  machines  engage  in  or  are  in  control  over  (i.e.,  information  processing)  should  be 
considered  IAS  design. 

2.  The  approach  to  automation  and  adaptation  should  be  more  flexible  and  dynamic  rather  then 
fixed.  This  suggests  a  more  flexible  system  or  interface  that  changes  based  on  some  criterion. 
Such  changes  can  occur  across  various  levels  of  control  and  roles,  and  should  be  based  on 
operator  needs  that  emerge  from  empirical  testing.  For  example,  an  operator  could  be 
provided  with  an  adaptive  interface  until  the  operator  is  able  to  understand  the  benefits,  and 
then  later,  the  operator  could  be  given  more  control  over  certain  roles,  or  be  made  completely 
adaptable. 

3.  Roberts  (2006)  is  careful  to  point  out  that  that  by  dynamic,  it  is  not  implied  that  it  should  be 
‘flexible’  per  se.  This  intentional  dynamic  flexibility  will  allow  the  operator  to  be  kept  in  the  loop 
while  still  allowing  a  relief  of  workload.  Artificial  intelligence  is  not  allegedly  ‘intelligent’  enough 
at  the  present  time  to  flexibly  attend  to  every  need  and  goal  of  the  present  operator.  The 
dynamic  nature  of  an  adaptive  interface  should  be  intentional;  that  is,  intentional  in  that  the 
way  the  interface  changes  should  be  tested  empirically  and/or  through  usability.  In  other  words 
the  adaptive  interface  should  not  rely  on  artificial  intelligence  (i.e.,  agents)  to  dynamically  and 
flexibly  adapt  to  an  operator’s  every  need,  but  an  interface  that  is  dynamic  based  on  pre¬ 
defined  operator  needs  should  be  provided. 


Kaber,  D.B.,  Wright,  M.C.,  Prinzel,  L.J.,  and  Clamann,  M.P.  (2005).  Adaptive  automation  of 
human-machine  system  information-processing  functions.  Human  Factors,  47(4),  730-741. 


Overview: 

This  paper  explores  the  ability  of  human  operators  to  interact  with  adaptive  automation  applied  to 


147 


various  stages  of  a  complex  system’s  information  processing,  and  is  defined  in  a  model  of  human- 
automation  interaction.  A  study  examined  human  performance  (adaptation)  with  adaptive 
allocation  of  automation  (control  mode  switching)  at  various  levels  of  information  processing  (from 
low  to  high  cognitive  processing).  The  means  of  adaptation  was  based  on  a  user  model  of  operator 
performance  on  a  secondary  monitoring  task  as  an  indication  of  workload. 

The  following  results  were  found: 

•  Participants  adapted  better  to  adaptive  automation  (AA)  when  it  was  applied  to  sensory  and 
psychomotor  functions  (i.e.,  information  acquisition  and  action  implementation),  than  to  AA 
applied  to  cognitive  functions  (i.e.,  information  analysis  and  decision  making). 

•  Adaptive  automation  (i.e.,  at  all  stages  of  information  processing  and  the  initiation  of 
automation)  was  superior  to  complete  manual  control  of  tasks. 

•  In  lower  cognitive  functions,  transparency  of  the  system  was  easier  to  maintain.  It  was  harder 
to  validate  complex  decisions  that  require  mental  simulation  and  calculations. 

•  Operator  performance  was  best  under  automation  under  all  roles,  except  for  action 
implementation  when  compared  to  full  manual. 

Conclusions  for  IASs: 

1 .  Study  results  suggest  that  knowledge  development  of  a  system  requires  sufficient  training  to 
ensure  that  an  operator  has  acquired  a  proper  mental  model  of  the  system. 

2.  There  is  a  need  to  investigate  the  possibility  of  sensory  cuing  for  adaptive  automation  (mode 
switching). 

3.  Adaptive  automation  may  be  better  applied  to  lower  (information  acquisition  and  action 
implementation),  rather  than  higher  information  processing  (analyzing  information  and  decision 
making). 

4.  Automated  information  acquisition,  analysis  and  action  implementation  can  help  reduce 
workload  and  increase  performance  in  ATC  tasks. 


Schneiderman  and  Maes  (1997).  Excerpts  from  debates  at  IUI  97  (Intelligent  User  Interface 
Conference)  and  CHI  97 


Overview: 

This  paper  is  an  excerpt  from  debates  at  IUI  97  (Intelligent  User  Interface  Conference)  and  CHI  97. 

Schneiderman  and  Mares  debate  direct  manipulation  versus  intelligent  agents.  Schneiderman, 

while  not  against  the  use  of  intelligent  agents,  cautions  in  their  implementation.  Maes  strongly 

advocates  for  the  use  of  agents. 

The  following  benefits  of  agents  are  identified  by  Maes: 

•  A  software  agent  has  the  ability  to  know  the  individual  operator’s  habits,  preferences,  and 
interests. 

•  A  software  agent  can  be  proactive. 

•  Software  agents  are  more  long-lived.  They  keep  running,  and  they  can  run  autonomously  while 
the  operator  goes  about  and  does  other  things. 

•  Software  agents  can  be  adaptive  in  that  they  track  the  operator’s  interests  as  they  change  over 
time. 
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Conclusions  for  IASs: 

1 .  Maes  claims  that  agents  can  be  used  to  help  operators  deal  with  increasingly  complex  and 
dynamic  environments,  such  as  the  internet  with  a  vast  network  that  is  continuously  changing. 

2.  Maes  claims  that  agents  can  be  used  to  maximize  operator  attention  and  the  time  taken  on 
other  tasks.  For  instance,  an  agent  can  monitor  the  environment  for  aspects  of  interest  rather 
than  the  operator  constantly  monitoring  and  not  spending  time  on  other  tasks. 

3.  Maes  identifies  several  misconceptions  about  agents: 

o  Agents  should  not  be  used  as  a  substitute  for  direct  manipulation.  Direct  manipulation 
interfaces  and  agents  can  be  complementary. 

o  Some  people  believe  that  agents  are  personified  or  anthropomorphized,  while  most 
agents  are  not. 

o  Another  misconception  is  that  agents  rely  on  traditional  Al  techniques,  like  knowledge 
representation  and  inferencing.  However,  most  agents  commercially  available  have 
proven  successful  with  large  numbers  of  operators  relying  on  either  operator 
programming  or  on  machine  learning  rather  than  traditional  Al  techniques. 

4.  Schneiderman  is  careful  to  point  out  that  agent-operator  collaboration  can  only  be  successful  if 
the  operator  can  understand  and  trust  the  agent.  Operators  must  be  able  to  turn  over  control  of 
tasks  to  agents  but  operators  must  never  feel  out  of  control.  He  cautions  designers  against  the 
use  of  anthropomorphic  representations  (e.g.,  Microsoft’s  paper  clip)  as  it  may  interfere  with 
predictability,  reduce  operator  control,  and  may  undermine  the  operator’s  responsibility.  He 
recommends  an  “invisible”  or  transparent  agent  as  potentially  more  effective. 

5.  Schneiderman  mentions  that  OMIs  should  be  predictable,  so  that  operators  trust  them.  He 
claims  that  direct  manipulation  designs  can  promote  rapid  learning,  as  they  support  rapid 
performance  and  low  error  rates  while  supporting  exploratory  usage  in  positive  ways. 


Reference: 


Hou,  M.  Kobierski,  R.,  Herdman,  C.  (2006).  Design  and  Evaluation  of  Intelligent  Adaptive  Operator 
Interfaces  for  the  Control  of  Multiple  UAVs.  Proceedings  of  the  RTO  Human  Factors  and  Medicine 
Panel  Symposium  held  in  Biaritz,  France. 


Overview: 

This  paper  reports  on  a  multi-phase  project  to  investigate  the  potential  of  artificial  intelligence  for 
the  control  of  multiple  UAVs.  The  three  phases  include  IAI  concept  development,  interface 
prototyping,  and  experimentation.  Human-in-the-loop  trials  in  a  realistic  mission  scenario  were 
conducted  to  examine  the  performance  model  developed  by  DRDC. 

Several  recommendations  are  provided  for  the  design  and  implementation  of  lAls.  This  paper  has 
been  previously  described  in  Frameworks  Section  8.4.4. 

Conclusions  for  IASs: 

1 .  Results  suggest  that  operators  of  lAls  should  be  given  a  training  period  before  actually  using 
the  system,  particularly  in  life-critical,  mission-critical  systems. _ 
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2.  A  hybrid  adaptive  OMI  based  on  experience  with  the  adaptive  system  may  increase  an 
operator’s  understanding  of  the  system  and  its  impact;  a  phase  dependent  mix  between  fully 
automatic  and  operator-controlled  adaptation. 

3.  The  system  should  inform  the  operator  of  any  interface  changes.  For  instance,  the  IAI  should 
either  indicate  for  a  few  seconds  where  it  is  going,  or  indicate  what  has  changed. 

4.  The  interface  should  allow  the  operator  to  return  to  the  system  state  that  was  in  effect  before 
the  IAI  reconfigured  the  display  to  increase  the  sense  of  operator  control. 

5.  The  design  of  each  intelligent  agent  in  a  rapid  prototype  operator  interface  should  be  based  on 
reality. 

6.  Intelligent  agents  should  be  made  aware  of  the  state  of  the  world  by  accessing  data  fusion 
interim  variables  and  associated  probabilities.  The  authors  suggest  that  this  would  allow  the  IAI 
to  produce  strategies  that  “play  the  odds”. 


Reference: 


Scallen,  S.F.  and  Hancock,  P.A.  (2001).  Implementing  Adaptive  Function  Allocation.  The 
International  Journal  Of  Aviation  Psychology,  11(2),  197-221 


Overview: 

This  paper  details  a  study  that  examined  the  efficacy  of  adaptive  allocation  on  operator 
performance  and  workload.  Adaptive  allocation  was  implemented  in  a  multiple  task  aviation 
paradigm.  Pilot  performance  was  evaluated  in  three  tasks  related  to  tracking,  system  monitoring, 
and  target  identification. 


Conclusions  for  IASs: 

1 .  Adaptive  function  allocation  (AFA)  appears  to  improve  tracking,  monitoring  and  targeting 
performance,  and  more  accurate  perception  of  the  passage  of  time  (or  increase  situational 
awareness). 

2.  Implementation  of  adaptive  allocation  of  automation  could  produce  positive  benefits  to  a  wide 
range  of  pilot  functions  including  task  prioritization,  mission  segmenting,  task  initiation  and 
cessation,  risk  identification,  and  workload  management. 

3.  The  authors  suggest  that  it  would  be  a  mistake  to  consider  automation  as  environment 
specific.  Therefore,  if  adaptive  automation  is  implemented  in  seemingly  disparate 
environments,  it  is  likely  that  both  the  physical  environment  and  the  human-machine 
interaction  will  co-evolve. 

4.  Performance  trade-offs  may  hinder  effective  implementation  of  adaptive  strategies. 

5.  Adaptive  aiding  could  help  prioritize  functions  (e.g.,  in  this  time-stress  phase,  pilots  restrict 
their  sampling  of  information  to  what  they  perceive  as  most  important,  but  they  are  not  always 
accurate). 
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Hancock,  P.A.,  Scallen,  S.F.,  and  Duley,  J.A.  (1995).  Interface  design  for  adaptive  automation 
technologies.  Proceedings  of  the  3rd  international  workshop  on  human-computer  teamwork 
(Human-Electronic  Crew:  Can  we  trust  the  team?).  Cambridge,  UK,  27-30  September  1994. 


Overview: 

The  paper  outlines  a  study  where  the  effects  of  differing  levels  of  pilot  involvement  were  examined 
in  the  context  of  initiating  automation.  The  following  levels  of  automation  were  investigated: 
complete  pilot  control,  complete  automation  control,  system-recommended  automation,  and 
system-invoked  automation.  Results  indicate  that  system  invoked  automation  produced  less  time 
in  manual  control,  less  time  to  initial  automation  (i.e.,  participants  were  transitioned  to  automation 
much  more  quickly  than  the  other  groups),  and  an  increase  in  fatigue. 

Conclusions  for  IASs 

1 .  The  authors  suggest  that  based  on  their  study,  the  strategy  to  initiate  automation  should 

involve  a  dynamic  model  of  adaptive  allocation  In  terms  of  context  dependency,  there  could  be 
the  following  allocation  of  tasks  to  the  agent:  complete  pilot  control,  complete  automation 
control,  system-recommended  automation,  and  system-invoked  automation  amongst  varying 
levels  of  initiation. 


Findlater,  L.,  &  McGrenere,  J.  (2004).  A  comparison  of  static,  adaptive,  and  adaptable  menus. 
Proceedings  of  ACM  CHI  2004,  pp.  89-96 


Overview: 

A  study  comparing  three  menu  conditions,  static,  adaptable  and  adaptive  was  performed.  Each 
menu  was  implemented  as  a  split  menu  but  differed  in  the  way  the  customization  was 
implemented.  Results  indicate  that  the  static  menu  was  significantly  faster  than  the  adaptive  menu, 
and  the  adaptable  menu  was  found  to  be  significantly  faster  than  the  adaptive  menu.  The  majority 
of  operators  preferred  the  adaptable  menu. 

There  are  two  main  approaches  to  the  personalization  of  interfaces  to  individual  operators: 
adaptive  interfaces  dynamically  adjust  the  interface  in  a  way  that  is  intended  to  support  the 
operator  (system  controlled).  By  contrast,  adaptable  interfaces  provide  customization  mechanisms 
but  rely  on  the  operator  to  use  those  mechanisms  to  do  the  adaptation  (operator  controlled). 

The  authors  identify  that  there  is  some  debate  in  the  HCI  community  (and  HF)  as  to  which  of  the 
two  approaches  (adaptive  or  adaptable  interfaces)  is  best.  One  side  argues  that  easy  to  use 
predictable  mechanisms  should  be  provided  to  keep  operators  in  control  of  the  system,  while  the 
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other  side  believes  that  if  the  right  adaptive  algorithm  can  be  found,  operators  will  be  able  to  focus 
on  their  tasks  rather  than  managing  their  tools  (high  argument  for  automative  control  in  HF 
domain). 


Conclusions  for  IASs: 

1 .  Results  suggest  that  easy-to-use  mechanisms  are  not  sufficient  for  effective  customization 
(adaptive);  examples  should  also  be  provided  to  operators  to  guide  them  on  how  to  use  the 
customization  feature. 

2.  Operators  value  an  interface  that  can  be  modified  to  suit  their  individual  needs. 

3.  Providing  operators  with  control  over  the  adaptation  of  their  interface  can  lead  to  better 
perceived  performance  and  higher  overall  satisfaction  (Note  that  this  result  indicates  that 
designers  have  to  be  careful  when  interpreting  operator  feedback  on  system  usability). 


Reference: 


Inagaki,  T.  (2006).  Design  of  human-machine  interactions  in  light  of  domain-dependence  of 
human-centered  automation  Cognition,  Technology  and  Work.  8,  161-167. 


Overview: 

This  paper  argues  for  multi-layered  “human-centered  automation”  by  taking  into  account  not  only 
enhancement  of  situation  awareness,  but  also  trading  of  authority  between  humans  and  machines. 

The  authors  define  “human-centered  automation”  as  an  approach  in  which  operators  and  systems 
collaborate  cooperatively.  They  also  argue  that  automation  can  be  domain-dependent  (e.g., 
“human-centered  automation  for  automobile”  can  be  quite  different  from  “human-centered 
automation  for  aviation  system”). 

Refer  to  this  article  for  concrete  examples  on  how  to  apply  human-centered  automation  in  aviation 
systems  and  automobiles. 

The  authors  propose  a  new  scale  of  automation  levels  (allocation  and  initiation)  to  Sheridan’s  list 

(1998)  that  includes  an  extra  level  (#6.5): 

1  The  computer  offers  no  assistance;  human  must  do  it  all 

2  The  computer  offers  a  complete  set  of  action  alternatives,  and 

3  Narrows  the  selection  down  to  a  few,  or 

4  Suggests  one,  and 

5  Executes  that  suggestion  if  the  human  approves,  or 

6  Allows  the  human  a  restricted  time  to  veto  before  automatic  execution,  or 

6.5  Executes  automatically  upon  telling  the  human  what  it  is  going  to  do,  or 

7  Executes  automatically,  then  necessarily  informs  humans 

8  Informs  him  after  execution  only  if  he  asks 

9  Informs  him  after  execution  if  it,  the  computer,  decides  to 

10  The  computer  decides  everything  and  acts  autonomously,  ignoring  the  human 

[From  Sheridan  (1992),  Inagaki  et  al.  (1998),  and  Inagaki  and  Furukawa  (2004)] 
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Conclusions  for  IASs: 

1 .  To  enhance  awareness  of  system  function,  the  interface  should  be  designed  to  allow  the 
operator  to:  (1)  understand  the  rationale  behind  the  initiation  of  automation;  (2)  recognize 
intention  of  the  automation;  (3)  share  the  situation  recognition  with  the  automation,;  and  (4) 
understand  the  system’s  limitations. 

2.  The  authors  propose  a  new  level  of  automation  to  Sheridan’s  list  (1998)  to  reduce  automation 
surprises  induced  by  an  automatic  action,  as  well  as  to  make  the  action  effective  in 
emergencies  (refer  to  the  scale  presented  in  the  above  Overview). 

3.  The  level  of  automation  should  be  adapted  to  the  situation. 

4.  System  designers  should  consider  the  possibility  of  the  system  to  initiate  automation 
autonomously  in  situations  where  there  are  little  resources  left  for  the  operator  to  give 
directives  to  the  system  (e.g.,  pilot  loses  consciousness). 


Solodilova,  I.,  and  Galster,  S.  (2006).  Information  optimization  for  the  UMV  operator  interface.  In 
Proceedings  of  RTO  Human  Factors  and  Medicine  Panel  (HFM)  Symposium  held  in. 


Overview: 

This  paper  details  the  Mind  Reference  framework  as  a  set  of  guidelines  and  instructions  for 
information  presentation  in  UMV  operator  interfaces.  The  theory  is  centred  on  how  pilots  in  flight 
use  a  variety  of  Rules,  Strategies  and  Structures  to  consolidate  data  from  many  sources  to  aid 
them  make  swift  and  accurate  and  significant  decisions. 

Mind  Reference  Framework: 

The  framework  consists  of  an  Information  Matrix  that  comprises  of  a  set  of  Rules,  Structures, 
Strategies,  and  Relationships.  The  concept  focuses  on  how  to  organise  information  throughout  the 
information-system.  It  also  helps  to  identify  and  explore  possible  presentation  modes.  The  concept 
analyzes  data  from  pilot’s  debrief  comments  and  through  their  experience  and  knowledge  (in  part 
gained  from  the  researcher  observing  and  following  parallel  flight  training)  to  uncover  how  pilots 
represent  information  cognitively.  This  data  is  then  recorded  in  the  matrix  to  be  consulted  as  a 
source  of  guidance  during  interface  design  for  organizing  information  and  how  to  present  that 
information.  A  set  of  step  principles  are  then  followed  for  the  design  of  the  interface  (framework  is 
presented  below). 
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STEP  -  PRINCIPLES 


MIND  REFERENCE  FRAMEWORK 


INFORMATION  MATRIX 


Environment 


Behavior 


Physical 


Prior 

Current 

Resultant/ 

Consequent 

and 

Intended 

-About  When...- 

TIME 


1 .  Format  information  to  be  consistent  with  the 
cognitive  demand 


2.  Identify  relevant  Rules,  Structures  and  Strategies 


3.  Organize  basic  information  emerged  through  £ 
w  l  Set  of  Rules,  Structures  and  Strategies 


4.  Represent  information  in  accordance  with  Matrix 


5.  Addition  relevant  information  emerged  through 
the  use  of  Matrix  to  be  incorporated 


6.  Group  complementary  task  relevant  information, 
to  minimize  the  need  for  searching 


/7.  Link  information  on  the  interface  and  throughout 

(  the  system  to  other  relevant  information  using 
\defined  relationships  established  in  an  acquired  set 


8.  Establish  and  indicate  meaningful  connections, 
associations  and  interdependencies  of  information 

All  measurements  to  be  represented  in 
comparison  and  relative  to  either  the  limit  or 
capadty _ 

10.  Represent  information  in  meaningful  units 
related  to  the  parameter  to  help  associate  and 
assimilate  information 


/'ll.  Minimize  routine  computations  by  associating 

(  related  information  and  representing  information  in 
\a  form  that  pilots  reference  it _ 

f  12.  Provide  a  holistic  overview  for  ease  of 
l  information  integration  and  association 


13.  Provide  a  detailed  overview  for  ee 
convergence  on  information  needed 


14.  Where  relevant  provide  information  about  the 
future  aircraft  state,  behaviour  and  performance 


Mind  Reference  Framework 

Please  refer  to  the  paper  for  details  (theory  and  implementation)  on  the  list  of  steps. 


£  o 
■§£ 


Conclusions  for  IASs: 

1 .  The  Mind  Reference  Framework  provides  a  useful  set  of  guidelines  and  instructions  on  how  to 
optimally  present  information  on  IAS  OMIs. 


Reference: 


Trouvain,  B.  and  Wolf,  H.L  (2002).  Design  and  Evaluation  of  a  Multi-Robot  Control  Interface.  In 
Proceedings  of  RTO  Human  Factors  and  Medicine  Panel  (HFM)  Symposium  held  in  Warsaw, 
Poland,  7-9  October  2002. 


Overview: 

This  paper  details  two  simulation-based  multi-robot  experiments  that  were  conducted  as  a  means 
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to  guide  and  support  the  development  of  a  multi-robot  control  interface.  It  was  determined  that 
robot  autonomy  is  required  for  a  multi-robot  system  to  be  managed  by  a  single  operator.  In 
separate  trials,  operators  had  to  manage  2,  4  and  8  robots  in  two  different  environments. 
Participants  controlled  navigation  and  monitored  inspection  of  robots  through  a  Ul.  The  two 
experiments  examined  different  levels  of  robot  autonomy  on  operator  performance.  Results  from 
these  two  simulations  guided  their  interface  design  process. 


Conclusions  for  IASs: 

1 .  Study  results  suggest  that  the  OMI  should  present  information  that  allows  optimal  performance 
when  monitoring  automation  (e.g.,  multiple  robots).  The  study  identified  that  optimal 
performance  occurred  with  five  robots.  Therefore,  the  layout  of  the  Ul  should  be  based  on  the 
requirement  to  display  status  information  of  four  to  five  robots  simultaneously,  at  a  maximum. 

2.  The  impact  of  autonomy  on  an  operator’s  performance  must  be  viewed  separately  for  the 
control  and  the  monitoring  aspect. 

3.  The  authors  believe  that  an  operator’s  ability  to  monitor  complex  systems  requiring 
autonomous  components  represents  the  actual  bottleneck  in  human  robot  teams.  That  is, 
without  sophisticated  operator  support  supervising  multi-robot  systems  larger  than  two  robots, 
it  is  difficult  to  realize  if  tight  monitoring  is  required. 


Kirlik,  A.,  Marked,  W.J.,  and  Kossack,  M.  (1992).  Comparison  of  display  enhancement  with 
intelligent  decision  aiding.  Technical  report:  School  of  Industrial  and  Systems  Engineering,  Georgia 
Institute  of  Technology,  Atlanta,  GA.  (NASA-CR-1 89895). 


Overview: 

Details  two  decision  aiding  strategies:  display  enhancement  and  intelligent  decision  aiding. 
Outlined  is  research  that  compares  and  contrasts  two  technologies  and  explains  the  interaction 
effects  introduced  by  the  different  skill  levels  and  different  methods  for  training  operators. 
Ecological  Interface  Design  has  been  proposed  as  an  abstraction  hierarchy  tool  to  illustrate  the 
functional  properties  of  a  system.  Research  suggests  that  novices  work  more  with  context-free 
elements  and  rules  and  are  not  as  able  to  identify  subtle  differences,  are  working  more  on  a  level 
which  can  be  best  described  using  rules.  In  contrast,  experts  behave  more  intuitively  and  are  very 
context  dependent. 


Conclusions  for  IASs: 

1 .  EID  can  be  used  as  a  hierarchy  tool  to  determine  the  functional  properties  of  a  system. 

2.  A  decision  aid  should  consider  the  cognitive  model  of  the  decision  maker. 

3.  An  intelligent  decision  aid  should  support  the  level  of  experience  and  skill  of  the  decision 
maker. 
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Overview: 

This  paper  discusses  the  cognitive  costs  and  benefits  related  to  automation  within  the  execution  of 
all  processes  that  lead  to  a  course  of  action  selection. 

The  authors  identify  that  the  largest  benefits  of  automation  relates  to  human  workload,  and  the 
reduced  demand  on  attentional  resources.  Automation  was  found  to  be  accompanied  by  major 
cognitive  costs,  mostly  related  to  operator  execution  of  a  task;  that  is,  when  operators  must  shift 
roles  from  monitor  (with  automation  turned  on)  to  active  agent  (automation  turned  off).  Further,  the 
passive  role  for  the  human  (as  monitor  of  automation)  was  found  was  found  to  potentially  prevent 
the  human  from  building  an  appropriate  mental  model  of  the  situation,  especially  for  the  recovery 
of  system  failures. 

The  authors  recommend  teaching  the  operator  to  become  an  effective  supervisor.  The  authors 
claim  that  this  can  be  an  effective  balancing  technique  between  reducing  mental  workload, 
attentional  demands,  the  effect  of  fatigue  and  stress  factors,  the  probability  of  errors,  and 
maintaining  situational  awareness. 


Conclusions  for  IASs: 


1 .  The  authors  recommended  investigating  several  factors  in  order  to  determine  the  level  of 
agent  role  and  authority  the  operator  should  have  within  a  mission  and/or  task  (the  level  of 
automation),  and  the  level  of  workload: 

a.  the  attentional  resources  required  from  the  human; 

b.  the  reduction  of  the  stress  and  fatigue  factors; 

c.  the  reduction  of  human  error  occurrence; 

d.  the  quality  of  situation  understanding  by  the  human; 

e.  the  human  capacity  to  recover  from  system  failure  or  the  occurrence  of 
unexpected  events;  and, 

f.  the  role  of  the  human  in  the  execution  of  the  overall  decision-making  task. 

2.  The  authors  advocate  training  the  human  to  adequately  supervise  the  system  functioning.  This 
can  offset  some  of  the  potential  costs  of  automation  by  decreasing  workload  and  enhancing 
situational  awareness.  The  operator  should  develop  an  appropriate  mental  model  of  the 
system;  training  would  ensure  that  the  operator  understands  how  the  system  is  working,  and 
that  operators  have  access  to  the  information  that  is  considered  by  the  automated  systems  in 
order  to  develop,  as  the  situation  is  evolving,  an  adequate  understanding  of  this  situation. 


Reference: 


Galster,  SM.,  Bolia,  R.S.  and  Parasuraman,  R.  (2002).  The  Application  of  a  Qualitative  Model  of 
Human-Interaction  with  Automation:  Effects  of  Unreliable  Automation  on  Performance.  In 
Proceedings  of  RTO  Human  Factors  and  Medicine  Panel  (HFM)  Symposium  held  in  Warsaw, 
Poland,  7-9  October  2002. 
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Overview: 


A  visual  search  paradigm  was  used  to  examine  the  effects  of  information  automation  and  decision- 
aiding  automation  in  a  target  detection  and  processing  task.  Specifically,  manual,  information 
automation,  and  decision-aiding  automation  conditions  were  investigated. 

Results  indicate  that  there  was  an  increase  in  correct  responses  and  a  reduction  in  search  times  in 
the  information  automation  cue  condition,  regardless  of  the  reliability  of  the  automation.  These 
results  suggest  that  this  is  most  likely  due  to  over-reliance  on  the  automation  to  give  the  correct 
guidance  resulting  in  an  “automation  induced  complacency  effect”  under  the  automatic  condition. 


Conclusions  for  IASs: 

1 .  Over-reliance  on  automation  can  result  in  an  automation  induced  complacency  effect. 

2.  Automated  information  cueing  can  improve  target  identification  performance  under  high  target 
density  conditions.  Thus,  automation  cueing  may  be  best  suited  for  complex,  dense 
information  environments,  when  an  operator  is  likely  to  be  already  near  a  peak  level  of 
workload. 

3.  Target  acquisition  and  action  implementation  where  there  is  lots  of  visual  noise  (i.e.,  a 
saturated  complex  visual  field)  is  enhanced  by  the  presence  of  a  reliable  IA  cue,  providing 
location  information. 


Reference: 


Horvitz,  E.  (1999).  Principles  of  mixed-initiative  user  interfaces.  In  Proceedings  of  ACM 
Conference  on  Human  Factors  in  Computing  Systems  (CHI'99),  pp.  155-166. 


Overview: 

The  authors  review  principles  for  directly  manipulating 
automation  and  machine  learning.  These  principles  are 
highlighted  in  terms  of  the  program  called  LookOut,  an 
automated  system  for  scheduling  and  meeting  management. 

LookOut:  Is  a  program  that  automatically  populates  meeting 
request  information  based  on  an  email  message  text  in  the 
body  and  subject. 

Initiation  of  Lookout:  The  system  can  be  initiated  either  by  the 
human  (clicking  on  icon  or  when  prompted  by  the  system)  or  automatically  by  the  system  based  on 
a  goal-based  user  model. 

Direct  manipulation.  The  operator  communicates  directly  with  the  system  through  an  animated 
widget. 

User  model.  The  user  model  is  based  on  a  “function  of  an  inferred  probability”  that  the  operator 
has  a  goal  of  performing  scheduling  and  calendaring  operations. 

Confidence  estimation.  The  level  of  automation  (initiation  and  action)  is  based  on  the  system’s 
uncertainty  of  the  operator’s  goals  which  is  based  on  the  user  model.  The  authors  applied 
probabilistic  models  of  an  operator’s  goals.  This  is  used  to  perform  real-time  inferences  about  the 
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probability  of  alternate  feasible  goals  by  monitoring  the  current  program  context,  and  the  operator’s 
sequence  of  actions  and  choice  of  words  used  in  a  query.  Bayesian  network  models  were  partially 
used  to  for  a  base  for  the  confidence  estimation  algorithms. 

Displaying  automation  uncertainty.  The  level  of  uncertainty  about  the  operator’s  goals  is  displayed 
to  the  operator  via  visual  indicators.  At  high  levels  of  certainty,  a  character  appears  and  indicates 
that  it  has  readied  a  calendar  view  to  show  the  operator  or  has  created  a  tentative  appointment 
before  displaying  the  results.  At  lower  levels  of  confidence,  LookOut  inquires  about  the  operator’s 
interest  in  either  seeing  the  calendar  or  scheduling  an  appointment,  depending  on  the  system’s 
analysis  of  the  message  being  viewed. 

Automated  tasks.  The  decision  of  initiating  automation  is  based  on  whether  an  agent  believes  it  will 
have  greater  expected  value  than  inaction  for  the  operator,  taking  into  consideration  the  costs, 
benefits  and  uncertainties  in  the  operator’s  goals.  Refer  to  the  paper  for  implementation  details. 

Timing  of  prompting  the  initiation  of  automation.  Automation  and  alerts  of  initiating  automation  is 
based  on  models  of  attention  that  consider  the  temporal  pattern  of  an  operator’s  focus  of  attention 
(timing  model). 

Machine  learning.  The  system  is  designed  to  continue  to  learn  from  operators  through  caching 
operator  behaviour  with  the  system  and  by  the  operator  specifying  a  policy  for  continual  learning 
(e.g.,  set  system  to  cache  behaviour  at  particular  times). 

The  authors  recommend  considering  several  critical  factors  when  implementing  integration  of 
automated  services  with  direct  manipulation  interfaces,  as  discussed  below. 


Conclusions  for  IASs: 

1 .  Uncertainty  about  an  operator’s  goals  can  provide  good  input  for  inferring  about  an  operator’s 
intentions  to  perform  an  operation.  Computers  are  often  uncertain  about  the  goals  and  the 
current  focus  of  attention  of  an  operator.  In  many  cases,  systems  can  benefit  by  employing 
machinery  for  inferring  the  uncertainty  about  an  operator’s  intentions  and  focus. 

2.  Considering  the  status  of  an  operator’s  attention  in  the  timing  of  services.  Systems  (or  agents) 
could  use  models  of  attention  and  consider  the  costs  and  benefits  of  deferring  action  to  a  time 
when  the  automation  will  be  less  distracting  to  the  operator. 

3.  Context-dependent  automation.  Automated  functions  should  be  applied  in  a  context-relevant 
manner  based  on  uncertainty  in  an  operator’s  goals  and  attention. 

4.  The  system  should  resolve  uncertainties  through  a  dialog  with  the  operator.  If  a  system  is 
uncertain  about  an  operator’s  intentions,  it  should  be  able  to  engage  in  a  dialog  with  the 
operator,  considering  the  costs  of  potentially  bothering  an  operator  needlessly. 

5.  Direct  invocation  and  termination  of  automation  should  be  provided.  Efficient  means  should  be 
provided  which  operators  can  directly  invoke  or  terminate  the  automated  services. 

6.  Operators  should  have  an  efficient  means  to  modify  automation  behavior.  Agents  should  be 
designed  so  that  operators  can  complete  or  refine  an  analysis  provided  by  an  agent. 

7.  Agent-operator  interaction  should  employ  socially  appropriate  behaviors.  An  agent  should  be 
designed  to  behave  in  a  way  that  matches  social  expectations. 

8.  Recent  operator  interactions  with  the  system  should  be  saved.  Systems  should  maintain  a 
memory  of  recent  interactions  with  operators  and  provide  mechanisms  that  allow  operators  to 
make  references  to  objects  and  services  included  in  “shared”  short-term  experiences. 

9.  Learning  by  observing  operator  behavior.  Systems  should  be  designed  so  that  they  continue  to 
learn  about  an  operator’s  goals  and  needs 
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Dzindolet,  M.T.,  Beck,  H.P.,  Pierce,  L.G.,  and  Dawe,  L.A.  (2001).  A  framework  of  automation  use. 
Army  Research  Laboratory  Technical  Report  ARL-TR-2412/. 


Overview: 

This  paper  discusses  how  future  decision  support  can  be  improved  by  understanding  the  causes  of 
successes  and  failures  of  past  decision  support  systems.  The  purpose  of  this  report  was  to  present 
a  general  framework  to  understand  an  operator’s  decision  to  allocate  tasks  to  automation. 

Two  studies  (Dzindolet  et  al.,  to  be  published;  Dzindolet  et  al.,  1999)  were  conducted  to  examine 
the  role  of  automation  bias  in  automation  disuse  and  misuse.  Results  indicated  that: 

•  When  automation  bias  played  a  role  in  the  decision  to  rely  on  automation,  misuse  occurred 
more  than  disuse. 

•  When  the  automation  bias  was  eliminated  (this  was  achieved  by  providing  automated 
decisions  only  after  operators  recorded  their  decision),  disuse  (not  misuse)  was  involved  in 
subsequent  task  allocation  decisions. 


Conclusions  for  IASs: 

•  The  framework  of  automation  use  presented  in  Dzindolet  et  al.,  (2001 )  framework  of 
automation  use  (2001)  predicts  that  cognitive,  motivational  and  social  processes  work  together 
to  cause  misuse,  disuse,  and  appropriate  automation  use,  and  may  be  useful  in  reducing 
automation  misuse  and  disuse.  Please  refer  to  the  article  for  a  detailed  description  of 
framework. 

•  According  to  the  general  framework,  disuse  appears  to  be  a  greater  problem  than  misuse.  For 
instance,  disuse  can  be  reduced  by  providing  operators  with  multiple  forms  of  feedback  of  the 
system’s  performance. 

•  The  authors  stress  that  automation  bias  should  be  controlled.  One  way  to  achieve  this  is  to 
provide  operators  with  the  systems’  decision  support  only  after  the  human  operator  has 
provided  a  decision. 
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Overview: 


The  authors  advocate  for  a  balanced  approach  to  authority  over  adaptation.  Authorship  should  be 
matched  to  the  abilities  of  the  agent,  operator  or  system.  For  instance,  giving  the  system  too  much 
or  too  little  responsibility  can  make  it  ineffective,  either  jeopardizing  the  mission  or  by  making  it  an 
annoyance.  Elementary  detection  theory  is  used  to  diagnose  conventional  detection  aids. 


Conclusions  for  IASs: 

The  authors  recommend  the  following  top-down  strategies  for  system  design: 

1 .  The  system  must  adhere  to  a  principle  of  balance  between  the  responsibilities  and  abilities  for 
each  detector,  operator  and  system  alike. 

2.  A  good  design  strategy  is  to  include  operators  in  the  entire  development  cycle,  whom  can  be 
consulted  to  confirm  the  choice  of  a  given  strategy  and  ensure  that  their  own  responsibilities 
are  commensurate  with  their  abilities. 

3.  The  design  strategies  in  this  paper  can  be  referred  to  for  the  development  of  target  detection 
systems. 
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Fernall,  A.P.  (1997).  Decision  support  solutions:  Analysis  of  reasons  for  past  successes  and 
failures.  DRA  Technical  Report  DRA/LS3/19K2.13/DOC.3/96/3/. 


Overview: 

This  paper  discusses  how  future  decision  support  can  be  improved  by  understanding  the  causes  of 
successes  and  failures  in  past  decision  support  systems.  Both  the  design  and  assessment 
processes  were  reviewed,  as  they  were  both  determined  to  be  a  fundamental  factor  in  the 
success/failure  of  a  support  system. 


Conclusions  for  IASs: 

1 .  The  authors  found  that  a  comprehensive  cognitive  task  analysis  can  guide  the  development  of 
a  task-relevant  decision  aid,  and  increase  the  depth,  breadth  and  focus  of  the  system. 

2.  The  authors  report  that  there  is  often  a  lack  of  appreciation  of  the  effects  of  introducing  new 
technology.  There  are  several  misconceptions  about  introducing  new  technology  including  that 
it  will  improve  decision  making,  that  operators  know  what  they  want,  and  that  introduction  of 
the  aid  will  lead  to  reduced  training  needs. 

3.  High  quality  HCI  is  often  lacking  in  the  design  and  implementation  of  many  systems. 

4.  The  authors  claim  that  project  assumptions  are  often  not  questions,  and  can  result  in  serious 
implications: 

a.  Critical  requirements  may  be  missed. 

b.  A  danger  that  functions  that  are  easy  to  automate  will  start  to  dominate  those  that 
are  not. 

5.  The  authors  found  that  the  traditional  design  approach  for  developing  military  computer _ 
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systems  is  fundamentally  unsuited  to  the  design  of  effective  decision  aids. 


Lock,  Z.,  Macklin,  C.,  and  Thompson,  D.  (2004).  Personalized  Briefing  Agents:  Phase  II  Technical 
Report  QINETIQ/KI/CIS/TR041 645/1 .0 


Overview: 

A  Personailzed  Briefing  Agent  (PBA)  system  was  developed  and  designed  to  interact  with  military 
officers  and  provide  them  with  updated  information  on  the  status  of  the  battle,  when  required,  in  a 
manner  suited  to  their  individual  requirements  (PBA  is  an  information  management  decision 
making  aid.  The  authors  discuss  the  application  of  information  processing  and  user  modeling  as  a 
means  of  controlling  information  overload,  and  also  to  increase  the  quality  of  decision  making.  In 
particular,  it  discusses  a  novel  user  modeling  method  to  design  a  system  that  provides  context- 
dependent  information  automatically. 

Novel  User  Modelling:  Information  about  operators  is  acquired  and  represented  so  that  it  can  be 
used  to  automatically  adapt  the  information  presented  to  the  operator  (what  the  authors  refer  to  a 
personalizing  the  computer  system).  A  multi-component  user  modeling  approach  is  used  to  apply 
adaptation.  Section  3.6  details  these  component  models.  Details  of  these  models  are  out  of  the 
scope  of  this  review. 


Conclusions  for  IASs: 

1 .  A  multi-component  rather  than  a  single-component  approach  to  user  modelling  is  used  to 
overcome  the  limitations  of  the  single  approach.  In  a  single  approach,  difficulties  in  applying 
automation  adaptation  can  arise  where  certain  operator  information  requirements  change  over 
time  and  others  stay  the  same  (e.g.,  a  military  decision  maker  that  changes  roles  within  a 
team). 

2.  Multi-component  approach:  Each  information  requirement  is  associated  with  a  particular 
operator  perspective,  such  as  an  operator’s  team  membership,  assumed  role,  or  current 
operation.  Each  perspective  has  a  set  of  requirements  and  is  equivalent  to  a  single-component 
user  model.  An  overall  user  model  will  then  consist  of  a  set  of  components.  Each  of  these 
components  has  an  associated  weight,  which  indicates  the  contribution  a  component  makes  to 
overall  relevance  to  the  operator. 

3.  When  a  perspective  changes,  the  corresponding  component  can  be  replaced  with  another  that 
is  more  appropriate  to  the  new  situation.  All  other  components,  unaffected  by  the  perspective 
change,  remain  in  the  overall  user  model. 
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Hilburn,  B.G.  (2002).  Evaluating  Human  Interaction  with  Advanced  Air  Traffic  Management 
Automation.  In  Proceedings  of  RTO  Human  Factors  and  Medicine  Panel  (HFM)  Symposium  held 
in  Warsaw,  Poland,  7-9  October  2002. 


Overview: 

This  paper  summarizes  the  results  of  two  studies.  The  first  study  examined  air  traffic  controllers’ 
attitudes  toward  possible  new  forms  of  automation.  In  the  second  study,  a  series  of  human-in-the- 
loop  simulations  were  conducted  to  evaluate  the  potential  benefits  of  advanced  Air  Traffic 
Management  (ATM)  automation  on  human-machine  system  performance. 


Conclusions  for  IASs: 

1 .  Results  suggest  that  operator  acceptance  is  heavily  dependent  not  only  on  perceived  reliability 
of  the  new  system,  but  also  on  the  nature  of  the  reliability  (e.g.  is  the  system  prone  to  false 
alarms?  misses?),  and  on  the  costs  involved  with  verifying  an  automated  system’s  functioning, 
as  compared  to  the  relative  costs  of  a  miss/false  alarm. 

2.  Ensure  a  proper  mental  model  of  the  system L  Knowing  what  information  the  system  is  using 
(and  not  using),  and  how  the  system  can  be  expected  to  behave  in  various  situations  is  critical 
for  the  development  of  operator  trust  in  an  automated  “partner.” 
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Overview: 

The  paper  outlines  a  study  to  investigate  complacency  effects  when  operators  interact  with  an 
automated  aid  in  a  process  control  simulation  task.  Possible  performance  consequences  (i.e. 
automation  bias  in  terms  of  commission  errors,  and  impairments  of  return-to-manual-performance 
in  case  of  automation  breakdown)  are  also  examined.  The  effect  of  a  specific  training  intervention 
to  reduce  complacency  by  exposing  participants  intentionally  to  automation  failures  is  also 
investigated. 

The  results  provide  clear  evidence  for  complacency  effects  due  to  insufficient  verification  of 
recommendations  provided  by  the  automated  aid. 


Conclusions  for  IASs: 

1 .  The  authors  suggest  that  confronting  operators  with  rare  automation  failures  during  training 
may  be  a  suitable  way  to  reduce  complacency,  yet  may  not  be  sufficient  to  prevent 
complacency  effects  completely. 
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2.  The  risk  of  commission  error  was  associated  with  comparatively  high  levels  of  complacency 
only;  usually  less  information  needs  to  be  sampled  to  falsify  an  automatically  generated 
diagnosis  than  to  verify  it  completely. 


Reference: 


Bennett,  Cress,  Hettinger,  Stautberg,  and  Haas  (2001).  A  Theoretical  Analysis  and  Preliminary 
Investigation  of  Dynamically  Adaptive  Interfaces.  The  International  Journal  of  Aviation  Psychology, 
11( 2),  169-195. 


Overview: 

A  study  to  examine  the  effects  of  2  different  adaptive  OMIs  (candidate  and  dynamic  adaptive 
interface)  on  routing  behaviour  was  performed  in  a  simulated  flight  task.  Non  traditional  controls  (a 
force  reflecting  stick)  and  displays  (a  configurable  flight  director)  were  developed  to  support  task 
performance  in  real  time.  Results  found  that  the  candidate  and  adaptive  conditions  led  to 
significant  performance  advantages  to  operator  performance  compared  to  the  standard  interface. 
There  were  no  significant  differences  found  in  performance  between  the  candidate  and  adaptive 
interfaces. 

The  authors  describe  a  dynamically  adaptive  interface  (DAI)  as  a  computer  interface  that  changes 
the  display  or  control  characteristics  of  a  system  (or  both)  in  real  time.  An  adaptive  interface  in  this 
study  is  referred  to  as  an  interface  that  allows  an  individual  to  modify  the  characteristics  of  an 
interface,  but  not  in  real  time.  There  are  three  major  categories  in  which  adaptations  in  the  DAI  can 
be  triggered:  changes  in  system  state,  human  performance  models,  and  on-line  assessment. 


Conclusions  for  IASs: 

1 .  The  candidate  and  adaptive  conditions  led  to  significant  performance  advantages  to  operator 
performance  compared  to  the  standard  interface.  There  were  no  significant  differences  found 
in  performance  between  the  candidate  and  adaptive  interfaces. 

2.  The  authors  suggest  that  operators  should  have  a  well-formed  mental  model  of  the  system 
(i.e.,  the  system  should  be  transparent  to  the  operator). 

3.  The  authors  outline  that  one  fundamental  challenge  in  designing  effective  DAIs  is  to 
demonstrate  that  the  dynamic  changes  in  display  or  control  information  do  not  interfere  with 
either  the  development  or  the  execution  of  skilled  behavior. 


Reference: 


Kirschenbaum,  S.S.  (2002).  Uncertainty  and  Automation.  In  Proceedings  of  RTO  Human  Factors 
and  Medicine  Panel  (HFM)  Symposium  held  in  Warsaw,  Poland,  7-9  October  2002. 
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Overview: 


This  paper  explores  a  range  of  issues  related  to  uncertainty  and  automation  and  suggests 
methods  to  mitigating  the  effects  of  uncertainty  associated  with  automation. 

The  authors  report  that  uncertainty  can  occur  in  any  situation  in  which  the  operator  does  have  a 
well-established  mental  representation  (or  mental  model)  of  the  “world”,  and  in  this  case,  the 
system.  Difference  between  experts  and  novices  are  explored.  The  authors  report  that  experts  are 
more  likely  to  be  aware  that  there  is  uncertainty  in  most  situations.  This  can  lead  to  not  trusting  the 
system  (e.g.,  a  generated  answer).  The  Representation  Match  Hypothesis  can  be  used  to  guide 
the  display  of  uncertainty  to  alleviate  these  problems.  It  states  that  processing  of  uncertainty  will  be 
most  effective  (least  errors,  least  time)  when  the  representations  of  uncertainty  match  the 
representations  required  for  problem  solving. 


Conclusions  for  IASs: 

1 .  Adaptive  automation  requires  the  system  to  be  transparent  so  that  the  operator  and  the  system 
can  function  as  a  team.  When  automation  behaves  in  an  unexpected  manner,  operators  tend 
to  attribute  it  to  a  breakdown  in  functioning  (hardware  or  software),  and  reject  all  subsequent 
output. 

2.  The  Representation  Match  Hypothesis  can  guide  the  display  of  uncertainty  by  matching  the 
representations  required  for  problem  solving. 


Reference: 


Taylor,  R.M.  (2002).  Capability,  Cognition  and  Autonomy.  In  Proceedings  of  RTO  Human  Factors 
and  Medicine  Panel  (HFM)  Symposium  held  in  Warsaw,  Poland,  7-9  October  2002. 


Overview: 


This  paper  is  a  review  of  various  frameworks  related  to  military  capability,  cognition  and 
automation,  and  their  relation  to  the  role  of  the  operator  within  a  system.  Only  cognition  and 
automation  will  be  reported. 

Cognition  Frameworks:  These  frameworks  are  useful  for  understanding  what  role  operators  play  in 
advanced  automated  systems.  Simple  perceive-decide-act,  and  the  air  combat-based  OODA  loop 
(observe,  orient,  decide,  act)  frameworks  are  recommended  for  function  allocation  to  an  agent 
(operator  or  a  system).  The  author  argues  for  an  alternative  class  of  frameworks  which  include 
“situated  cognition”,  “naturalistic  decision-making”,  and  “cognition  in  the  wild”.  These  frameworks 
consider  cognition  in  context  and  natural  situations.  This  approach  recognizes  that  operator 
performance  is  constrained  by  the  conditions  under  which  it  takes  place,  and  focuses  on  the 
variety  of  human  performance  and  what  cognition  does,  rather  than  what  cognition  is,  and  the 
internal  mechanisms  for  achieving  it. 

Automation  Frameworks:  The  author  describes  automation  as  a  means  of  replacing  skill-based 
behaviour  (well-defined,  highly  structured  domain),  replacing  and  supporting  some  rule-based 
behaviour,  and  as  supporting  some  knowledge-based  behaviour  (ill-defined,  unstructured  domain). 
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Conclusions  for  IASs: 

1 .  Analysing,  identifying,  and  allocating  functions  is  an  essential  early  step  in  HF  engineering. 
The  author  reports  however  that  standardizations  (e.g.,  STANAG  3994)  are  limited  to  simple 
function  analysis,  rather  than  defining  the  allocation  of  functions. 

2.  The  author  argues  for  the  use  of  frameworks  that  consider  cognition  in  context  and  natural 
situations.  This  approach  recognizes  that  operator  performance  is  constrained  by  the 
conditions  under  which  it  takes  place,  and  focuses  on  the  variety  of  human  performance  and 
what  cognition  does,  rather  than  what  cognition  is,  and  the  internal  mechanisms  for  achieving 
it. 

3.  The  authors  point  to  a  central  problem  in  adaptive  automation  which  is  related  to  the 
determining  whether  and  when  transfers  of  control  to  the  operator  should  occur.  One 
technique  proposes  that  the  transfer  should  occur  when  the  expected  utility  of  transfer  is 
greater  than  that  of  retaining  the  decision-making.  Another  option  is  in  highly  uncertain 
situations  where  the  agent  should  relinquish  and  transfer  control  to  the  operator. 


Reference: 


Goossens,  A.A.,  van  Ginkel,  H.T.A.,  Theunissen,  E.,  de  Vries,  M.F.L.,  Koeners,  G.J.M.,  Roefs, 
F.D.  (2006)  Exploring  autonomy  and  authority  issues  with  respect  to  conflict  prediction  and 
resolution.  In  Proceedings  of  RTO  Human  Factors  and  Medicine  Panel  (HFM)  Symposium  held  in 
Biaritz,  France. 


Overview: 

This  paper  describes  two  experiments  that  were  conducted  to  explore  the  level  of  authority  (LoA) 
influence  on  operator  Situational  Awareness  and  performance  within  a  ground  control  station 
simulation  for  UAVs 

Experiment  1  was  conducted  to  compare  the  influence  of  three  different  LoAs  on  several  different 
measures  of  performance  and  SA.  None  of  these  measures  yielded  significant  differences 
between  the  LoA  conditions.  Interestingly,  participants  often  did  not  have  good  enough  SA  to 
decide  whether  or  not  it  was  necessary  to  intervene,  and  if  so,  exactly  what  action  they  should 
undertake.  In  order  to  better  support  participants’  SA,  the  authors  re-designed  the  system  to  allow 
the  participants  to  intervene  timely  (i.e. ,  before  the  system  intervenes)  and  correctly. 

In  the  second  experiment,  only  two  different  LoAs  were  compared,  removing  the  highest  LoA  from 
consideration.  The  results  of  this  experiment  are  very  consistent  with  the  first.  Again,  significant 
differences  were  not  found  between  the  LoA  on  any  of  the  measures.  Participants  scored  well  on 
the  questionnaire  concerning  their  knowledge  and  understanding  of  the  situation,  in  both  LoA 
conditions. 


Conclusions  for  IASs: 

1 .  In  this  instance,  LoA  did  not  impact  Situational  Awareness. 


165 


5.2.1. 1  Adaptation  Cycles 

Adaptation  cycles  are  defined  as  the  frequency  with  which  automation  is  turned  on/off  over  a 
period  of  time.  There  is  a  continuum  of  short  to  long  cycles  of  adaptive  automation,  and  what 
constitutes  short  or  long  cycles  is  dependent  of  the  particular  task  being  performed. 

Allocation  from  the  human  to  the  system  for  brief  time  periods  could  produce  a  potentially 
uncontrollable  oscillation  of  manual  and  automated  control.  These  conditions  are  referred  to 
as  automation  cycling ,  which  is  measured  as  the  frequency  of  automation  change  over  a 
specific  time  period.  If  uncontrolled,  the  oscillation  could  prove  particularly  detrimental  to 
overall  performance.  Particularly,  short  episodes  of  automation  may  also  prove  so  detrimental 
that  the  operator  may  simply  shut  the  system  off.  Hence,  the  failure  to  understand  operator 
response  to  short  episodes  of  automation  might  obviate  the  fundamental  purpose  of  intelligent 
adaptive  systems. 

•  Excessively  long  or  excessively  short  adaptation  cycles  can  limit  the  effectiveness 
of  IASs  in  enhancing  operator  performance  (Hilburn  et  al.,  1993); 

•  Occasional  brief  reversions  to  manual  control  can  counter  some  monitoring 
inefficiency  typically  associated  with  long  cycle  automation  (Hilburn  et  al., 

1993); 

•  Supervision  of  dynamic  tasks  is  significantly  worse  than  the  supervision  of  more 
stable  tasks  (Parasuraman  et  al.,  1996);  and, 

•  Detection  of  automation  failures  is  substantially  degraded  in  systems  with  static 
automation  in  which  the  allocation  of  tasks  between  operator  and  system  remains 
fixed  over  time,  approximately  20  minutes  (Parasuraman  et  al.,  1996). 


5.2.1. 2  Dynamic  (Learning)  Adaptation 

The  dynamics  and  capabilities  of  an  IAS  could  also  change  over  time,  as  the  system  becomes 
more  familiar  with  the  operator  (e.g.,  interaction  style,  commonly  used  tasks,  etc.).  Section 

10.2.1.2  describes  a  variety  of  studies  relating  to  the  dynamic  (i.e.,  learning)  aspects  of  IASs. 


Reference:  rS3l  \  \ 

IBB 

Wolfman,  S.A.,  Lau  Pedro  Domingos,  T.  and  Weld,  D.S.  (2001).  Mixed  Initiative  Interfaces  for 
Learning  Tasks:  SMARTedit  Talks  Back.  In  proceedings  of  IUI’01 ,  January  14-17,  2001,  Santa  Fe, 
New  Mexico,  USA. 


Overview: 

An  interface  for  machine  learning  is  proposed.  The  paper  describes  a  variety  of  interaction  modes 
that  enhance  the  learning  process  and  presents  a  decision-theoretic  framework,  called  DIAManD, 
for  choosing  the  best  interaction. 

The  authors  propose  that  machine  learning  systems  should  closely  resemble  human  teacher- 
student  relationships  and  follow  the  example  of  the  proactive  yet  considerate  student.  For  instance, 
the  system  should  ask  questions,  propose  examples  and  solutions,  and  relate  its  level  of 
knowledge  when  appropriate  to  make  the  interaction  more  effective. _ 
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DIAManD  is  a  system  for  selecting  among  various  interaction  modes  using  a  multi-attribute  utility 
function.  The  interaction  modes  provide  a  variety  of  methods  for  an  operator  to  interact  with  the 
system.  The  system  selects  from  a  set  of  interaction  modes  the  mode  it  judges  most  appropriate 
based  on  the  attribute  vectors.  The  best  of  these  modes  is  presented  to  the  operator  and  controls 
the  next  stage  of  discourse,  updating  the  state  of  the  learner.  The  modes  are  then  rescored  based 
on  the  new  state  of  the  learner. 


Conclusions  for  IASs: 

1 .  The  operator  of  a  system  should  be  able  to  override  the  system's  choice  of  interaction 
mode  and  choose  a  mode  that  he/she  prefers. 

2.  An  attribute  set  must  reflect  the  balance  between  operator  effort  and  the  value  to  the  task 
and  system. 

3.  The  authors  recommend  five  appropriate  but  general  attributes,  each  of  which  should  be 
viable  for  most  learning  system  and  interaction  library  combinations.  The  attributes 
(operator  input,  level  of  continuity,  and  probability  of  correction)  focus  on  operator  effort 
and  represent  the  physical  and  mental  effort  required  from  an  operator.  The  attributes 
(task  progress  and  value  to  the  system)  focus  on  the  achievement  of  an  operator's 
objective.  These  measures  reflect  an  operator’s  typical  objective  of  a  machine  learning 
system,  which  is:  complete  the  task  by  refining  the  hypothesis  of  the  learning  system  until 
it  correctly  describes  the  data. 


Parush,  A.  and  Auerbach  (forthcoming).  Adaptive  User  Interfaces:  Examination  of  Adaptation 
Costs  in  User  Performance.  To  be  presented  at  the  First  International  Conference  on  Augmented 
Cognition  (part  of  Human  Computer  Interaction  International  2005),  Las  Vegas,  Nevada,  US,  July 
22-27,  2005 


Overview: 

Adaptive  and  mixed-initiative  interfaces  were  compared  in  a  simulated  web-based  help  desk.  The 
content  of  the  simulation  included  requests  from  customers. 

Methodology:  The  interface  could  change  in  the  way  requests  were  displayed  and  the  way  the 
requests  were  handled  in  terms  of  priority.  The  amount  of  incoming  requests  (task  load)  was  the 
trigger  for  adapting  the  interface  from  form-based  (designed  for  low  task  load)  to  list-based 
(designed  for  high  task  load).  The  adaptive  interface  changed  from  list  to  form  automatically,  and 
vice  versa,  when  a  critical  number  of  requests  were  reached.  The  adaptation  was  based  on  the 
amount  of  operator  workload.  The  mixed-initiative  interface  changed  from  list  to  form  based,  and 
vice  versa,  only  after  the  operator  had  read  and  accepted  a  recommendation  to  do  so.  The  two 
control  groups  received  either  a  static  form  or  static  list  based  interface. 

Results:  Findings  indicated  that  participants  in  the  mixed-initiative  interface  group  chose  to  ignore 
the  recommendations  even  though  it  came  with  a  cost  of  poor  performance,  perhaps  to  avoid  extra 
workload.  This  study  suggests  that  a  mixed-initiative  OMI  is  not  always  the  best  solution, 
particularly  for  situations  where  high  workload  is  involved.  The  operators  of  this  simulation  were 
expected  to  increase  their  already  high  workload  by  having  to  first  decide  whether  or  not  to  switch 
the  interface  type,  and  secondly  to  implement  the  action  chosen.  This  manual  implementation 
could  have  served  as  an  extra  task  and  hence  increased  workload.  Operators  did,  however, _ 
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quickly  learn  to  work  with  the  adaptive  interface  which  decreased  the  potential  performance  cost 
associated  with  the  adaptation  (e.g.,  increased  workload  and  decreased  task  performance). 

It  was  concluded  that  there  is  a  need  to  influence  an  operator’s  choice  of  interaction  strategy.  They 
recommend  making  the  operator  realize  the  longer-term  implications  of  the  adaptive  interface  and 
then  introduce  them  into  the  choice  situation.  In  order  to  achieve  that,  operators  may  be  given  at 
the  initial  phase  of  working  with  the  system,  a  fully  automatic  adaptation  system  to  help  them  learn 
the  longer-term  potential  benefits  of  interact  with  an  adaptive  interface.  Once  operators  are  at  the 
point  of  having  a  good  understanding  of  the  adaptation  and  its  impacts,  then  operator-controlled 
adaptation  can  be  introduced. 


Conclusions  for  IASs: 

1 .  Operators  of  systems  with  adaptive  interfaces  should  be  given  a  training  period  before  actually 
using  the  system,  particularly  in  life-critical,  mission-critical  systems.  That  is,  operators  should 
only  use  the  “real  system”  once  they  achieve  a  performance  level  that  minimizes  or  eliminates 
potential  costs  of  adaptation. 

2.  There  is  a  critical  need  for  operators  to  understand  and  experience  the  potential  benefits  of 
adaptation. 


3.  A  hybrid  adaptive  OMI  based  on  experience  with  the  adaptive  system  is  suggested  to  increase 
an  operator’s  understanding  of  the  system  and  its  impact:  a  phase  dependent  mix  between 
fully  automatic  and  operator-controlled  adaptation. 


Yoo,  J.,  Gervasio,  M.,  and  Langley,  P.  (2003).  An  adaptive  stock  tracker  for  personalized  trading 
advice.  Proceedings  of  the  International  Conference  on  Intelligent  User  Interfaces,  pp.  197 


Overview: 

The  Stock  Trader  system  investigated  operator  performance.  The  system  addresses  information 
overload  by  tailoring  recommendations  based  on  an  individual  operator's  investment  styles.  The 
system  utilizes  this  profile  to  rank  stocks,  and  it  revises  the  profile  based  on  traces  of  operator 
behavior.  The  system  automates  information  acquisition;  it  encompasses  sensing,  and  registers 
input  data. 

The  system  architecture  is  composed  of  the  following  elements: 

1 .  The  data  processing  unit  which  converts  raw  input  (i.e.,  current  stock  readings  and 
historical  trading  information)  into  reports  that  contain  buy  and  sell  recommendations  for 
the  operator.  It  relies  on  the  recommendation  module  to  make  appropriate  suggestions  for 
each  stock  based  on  individual  operator  profiles. 

2.  The  user  modeler  which  constructs  these  profiles  is  based  on  operator  responses  to 
previous  recommendations  (implicit). 

3.  The  information  manager  records  traces  of  an  operator’s  interactions  with  the  system  and 
also  maintains  awareness  of  operator  portfolios. 

4.  The  communication  unit  manages  the  information  into  and  out  of  the  server. 

5.  A  client  contains  a  communication  unit  and  a  graphical  user  interface  component. 

Results  from  a  study  conducted  with  novice  stock  traders  indicated  that  as  the  system  learned 
through  interaction  with  the  operator’s  past  behaviour,  the  traders’  acceptance  of _ 
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recommendations  increased.  Furthermore,  as  the  traders’  began  to  better  understand  how  the 
system  operates,  they  also  began  to  accept  more  recommendations. 

Conclusions  for  IASs: 

1 .  An  implicit  user  model  can  be  an  effective  and  non-obstructive  means  of  constructing  a  user 
model. 

2.  A  learning  system  can  improve  decision  support. 


Sears,  A.  and  Schneiderman,  B  (1994).  Split  menus:  Effectively  using  selection  frequency  to 
organize  menus.  ACM  Transactions  on  Computer  Human  Interaction,  1(1),  pp. 27-51. 


Overview: 

This  paper  describes  the  concept  of  Split  Menus,  a  means  of  managing  information  overload.  In 
split  menus,  the  most  frequently  accessed  items  are  located  above  the  top  partition  or  "split",  and 
the  other  items  are  located  below  the  split.  Split  menus  were  implemented  and  tested  in  two  in  situ 
usability  studies  and  a  controlled  experiment.  In  the  usability  studies,  performance  times  were 
reduced  by17%  to  58%  depending  on  the  site  and  menus.  In  the  controlled  experiment,  split 
menus  were  significantly  faster  than  alphabetic  menus  and  yielded  significantly  higher  subjective 
preferences.  It  is  a  fully  automatic  system  that  does  not  require  operator  input  for  the 
implementation  of  adaptation. 


Conclusions  for  IASs: 

1 .  Results  suggest  that  although  automatically  adapting  the  Ul  to  the  implicit  user  model 
improved  performance,  the  user  model  may  not  apply  to  all  operators  (e.g.,  different  usage 
patterns);  it  may  be  more  effective  to  develop  a  method  of  providing  an  operator  a  choice 
between  a  customizable  (adaptable)  Ul  to  give  them  the  option  to  turn  off  the  automatic 
feature. 

2.  Results  suggest  the  need  for  an  adaptive  split  menu  (or  interface)  where  the  interface 
dynamically  adapts  based  on  a  usage  pattern  (user  model). 


Reference: 


Franklin,  D.,  Budzik,  J.,  and  Hammond,  K.  (2002).  Plan-based  Interfaces:  Keeping  Track  of  User 
Tasks  and  Acting  to  Cooperate.  In  IUI’02,  January  13-16,  2002. 


Overview: 

This  paper  describes  the  concept  of  an  Intelligent  Classroom,  which  consists  of  a  computer  system 
that  dynamically  adapts  to  operator  actions  and  inputs  (gesture  and  voice)  in  a  classroom 
environment  (i.e.,  controls  camera,  automatic  presentation  slide-switcher).  The  algorithms  are 
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goal-based  and  driven  by  task  recognition. 

Intelligent  Classroom:  The  1C  is  an  automated  lecture  facility  prototype  that  serves  as  its  own 
audio/visual  assistant.  The  operator  (e.g.,  speaker),  provides  a  presentation,  and  the  Classroom 
watches  and  listens,  and  when  appropriate,  assists  will  provide  assistance.  The  1C  keeps  track  of 
various  activities  pursued  by  the  speaker  as  well  as  its  own  activities  in  control  of  its  various 
autonomous  components. 

The  representation  is  used  three  ways  to  accomplish  a  goal:  plan  execution  (execute  a  plan  to 
achieve  a  goal),  plan  recognition  (match  the  operator’s  actions  to  a  set  of  known  plans),  and 
projection  (follow  the  operator’s  plan  and  project  future  actions).  A  set  of  agents  are  used  to 
monitor,  recognize,  and  execute  some  plan  to  accomplish  an  operator  goal. 

The  system  is  based  on  the  principle  that  the  world  is  composed  of  a  series  of  processes.  A 
process  is  a  single  agent  that  executes  a  sequence  of  actions.  It  is  composed  of  one  or  more 
discrete  steps,  each  of  which  specifies  a  number  of  continuous  actions  and  a  number  of  discrete 
events.  The  processes  are  designed  such  that  the  Classroom  can  essentially  use  the  same 
algorithm  for  executing  a  process  that  it  used  for  observing  the  operator  as  the  operator  executes  a 
process. 

To  alter  the  algorithm  so  that  the  Classroom  can  observe  the  operator  and  to  follow  along  with  the 
operator’s  plans,  only  a  portion  of  the  first  step  needs  to  be  changed.  Rather  than  performing  the 
primitive  actions  that  are  a  part  of  the  step,  the  Classroom  performs  “observation”  actions  that 
complement  the  primitive  actions. 

The  Process  manager  continually  steps  through  its  set  of  processes  to  keep  them  synchronized 
with  the  operator  and  revises  the  set  of  processes  when  required. 

Human-machine  cooperation.  The  operator,  in  executing  part  of  a  plan,  expects  the  Classroom  to 
do  its  part  of  the  plan.  A  plan  is  a  set  of  processes  (often  to  be  executed  by  a  number  of  different 
agents)  that  when  executed  together  successfully,  accomplish  some  goal.  In  the  Classroom,  most 
plans  have  one  process  executed  by  the  operator  and  one  or  two  processes  executed  by  the 
Classroom.  This  definition  makes  explicit  the  presence  of  other  agents  or  exogenous  events.  In  the 
Classroom,  these  plans  attempt  to  express  a  common  understanding  of  how  a  speaker  and  an 
audio/visual  assistant  interact. 


Conclusions  for  IASs: 

1 .  The  more  the  system  understands  its  operators  and  their  tasks,  the  more  useful  the  system 
will  be. 

2.  The  same  techniques  implemented  in  the  Intelligent  Classroom  can  be  applied  to  a  broad 
range  of  interactive  applications.  Refer  to  the  paper  for  details  on  how  to  implement 
techniques. 

3.  The  system  should  understand  the  operator’s  actions  in  the  context  of  what  it  believes  the 
operator  is  doing. 

4.  The  ability  to  provide  reason  to  the  operator’s  activity  is  crucial  to  the  implementation  of  an 
intelligent  operator  interface. 

5.  Plan  generation  and  recognition  are  a  promising  means  of  adaptive  automation  and 
estimating  pilot  intent. 

6.  Human-machine  cooperation  can  be  achieved  by  allowing  an  operator,  when  executing  a  part 

of  a  plan,  to  expect  a  system  to  help  in  executing  that  part  of  the  plan.  A  plan  is  a  set  of 
processes  (often  to  be  executed  by  a  number  of  different  agents)  that  when  executed 
together  successfully,  accomplish  some  goal.  Plans  attempt  to  express  a  common _ 
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understanding  of  how  a  speaker  and  an  audio/visual  assistant  interact. 


Sofge,  D.,  Bugajska,  M.,  Adams,  W.,  Perzanowski,  D.,  and  Schultz,  A.  (2003).  Agent-based 
Multimodal  Interface  for  Dynamically  Autonomous  Mobile  Robots,  Technical  Report  prepared  for 
Navy  Center  for  Applied  Research  in  Artificial  Intelligence,  Naval  Research  Laboratory, 
Washington, DC. 


Overview: 

An  agent-based  multi-modal  interface  is  presented  that  was  designed  as  a  means  for  the 
robot/operator  to  request  information  through  a  “natural  language  interface”  that  uses  combined 
speech  recognition  and  a  gesture  interpretation  process,  among  other  command  input  modes.  The 
dynamic  allocation  of  tasks  is  based  on  a  goal/spatial  relation  architecture. 

The  authors  define  “Human-centric”  as  a  system  that  focuses  on  the  needs  and  natural  modes  of 
interaction  of  the  operator  rather  than  the  robot.  A  key  feature  of  the  interface  is  the  use  of  multiple 
overlapping  (and  sometimes  redundant)  modes  of  communication  between  the  operator  and  the 
robot.  These  are  overlapping  (and  sometimes  redundant)  modes  of  communication  that  provide 
the  operator  with  a  natural  interface  to  the  system,  allowing  the  operator  to  choose  the  mode  of 
communication  most  comfortable  to  him/her  given  the  current  task,  situation  and  environmental 
conditions.  To  control  the  robots  through  the  autonomous  robot  agent,  the  operator  interfaces  with 
the  Robot  Interface  Client. 

Agents  provide  a  natural  and  flexible  means  for  integrating  multiple  interface  modules  together. 


Conclusions  for  IASs: 

1 .  Agent-based  architecture  can  provide  a  natural  and  scalable  approach  to  implementing  a 
multimodal  interface  to  control  mobile  robots  through  dynamic  automation. 

2.  Direct  communication  with  an  agent  through  an  interface  (i.e.,  natural  language  and  gestures) 
can  be  an  effective  means  of  human-machine  communication. 


Funk,  H.B.,  and  Miller,  C.A.  (1997).  Context  sensitive  interface  design.  Proceedings  of  the 
International  and  Interdisciplinary  Conference  on  Modeling  and  Using  Context  (CONTEXT-97),  Rio 
de  Janeiro,  Brazil,  February  4-6,  Federal  University  of  Rio  de  Janeiro  Ed.,  pp.  303-318 


Overview: 

The  paper  argues  that  three  elements  must  exist  to  support  effective  context  sensitive  interfaces: 
1)  the  ability  to  accurately  monitor  the  context;  2)  allows  the  operator  to  modify  the  control  and 
display  configuration;  and  3)  autonomous  configuration  changes  within  the  interface.  The  paper 
discusses  the  aspects  of  context  which  are  necessary  to  perform  interface  adaptation.  A _ 
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framework  is  provided  which  represents  context  characteristics  using  a  vocabulary  based  on  tasks 
and  goals  as  the  foundation  of  context  representation  and  tracking.  The  RPA  project  is  used  as  an 
example  of  this  framework. 


Conclusions  for  IASs: 

1 .  The  authors  recognize  that  tasks  can  serve  as  an  “appropriate  aggregation  tool  for  collecting 
information  requirements”.  The  authors  advocate  a  general  approach  which  entails  that  these 
requirements  are  captured  by  task,  and  then  are  summed  across  tasks  to  form  the  dynamic 
interface  configuration. 

2.  The  authors  identify  several  benefits  for  task-based  context  representations: 

•  Direct  mapping  between  human  actions  and  the  interface; 

•  Unsuccessful  and  ill-supported  tasks  can  demonstrate  what  upgrades  and  additions  to  the 
interface  or  the  automation  should  be  made;  and, 

•  Tracking  tasks  which  involve  multiple  individuals  on  the  team,  and  particularly  those  which 
involve  explicit  communication  between  team  members  can  directly  support  team 
collaboration. 


5.2.2  Principles  of  Interaction 

A  number  of  factors  enhance  or  inhibit  the  use  and  acceptance  of  Intelligent  Adaptive 
Systems.  The  principles  outlined  in  Sections  10.2.2.1  through  10.2.2.6  illustrate  potential 
design  features  necessary  to  improve  acceptance  and  utilisation. 

5.2.2. 1  Adaptive  Automation 

The  following  guidelines  for  the  development  of  adaptive  automation  are  suggested: 

•  Operators  often  expect  IASs  to  perform  the  task  at  least  as  well  as  them.  Thus,  any 
IAS  incorporated  into  the  cockpit/vehicle  must  be  highly  effective  to  be  accepted 
(Morris  and  Rouse,  1986); 

•  It  is  important  that  the  operator  can  initiate  the  adaptation,  even  if  this  is  normally 
done  by  the  IAS  (Noah  and  Haplin,  1986); 

•  It  is  important  that  the  operator  feels  4 in  control’,  even  when  the  IAS  is  performing 
tasks  (Morris,  Rouse  and  Frey,  1986); 

•  The  operator  must  not  be  confused  by  the  intervention  of  the  IAS,  especially  when  the 
operator  does  not  initiate  the  intervention  (Lehner,  Cohen,  Thompson  and  Laskey, 
1987); 

•  To  avoid  rapid  changing  of  the  task  allocation  between  the  operator  and  the  IAS,  a 
solution  is  to  have  the  IAS  initiate  the  off-loading  of  the  task,  and  the  operator  initiate 
the  re-capture  of  the  task  (Rouse,  Geddes  and  Curry,  1987);  and, 

•  The  use  of  complex  adaptive  systems  may  also  require  the  operator  to  learn  new  skills 
or  re-learn  familiar  tasks.  The  ability  of  an  operator  to  adapt  to  the  new  demands  of 
using  such  a  system  will  depend  on  the  understanding  of  the  functions  and 
capabilities  of  the  IAS  (Morris  and  Rouse,  1986;  Lehner,  Cohen,  Thompson  and 
Laskey,  1987). 
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5.2.2.2  Adaptive  Interface 

Intelligent  Adaptive  Systems  have  important  implications  for  the  OMI.  Noah  and  Haplin, 
(1986)  suggest  guidelines  for  the  design  of  IAS  OMIs: 

•  An  IAS  should  reduce  cognitive  workload  by  sharing  the  initiative  for  operator- 
system  dialogue.  The  IAS  should  volunteer  information  to  the  operator  and  make 
responses  appropriate  to  the  operator’s  intent; 

•  The  IAS  should  support  the  operator  by  monitoring  input  data  for  situations  that 
indicate  that  the  operator  should  be  alerted  and/or  which  suggest  significant  changes 
in  the  situation.  The  operator  should  be  able  to  specify  what  these  conditions  might 
be; 

•  An  IAS  system  should  present  information  with  progressively  more  detail  as  the 
situation  demands  and  the  operator  should  be  able  to  control  this  function.  The 
problem  of  data  overload  can  be  alleviated  by  the  IAS  presenting  only  summarised 
information  first,  and  then  elaborating  when  demanded  by  the  operator; 

•  The  operator  should  be  able  to  rapidly  reconfigure  the  IAS  in  response  to  changes  in 
goals  and  problem-solving  strategies; 

•  Displays  and  controls  should  operate  within  a  metaphor  that  is  consistent  with  the 
operator’s  conceptual,  or  ‘mental’,  model.  In  other  words,  the  IAS  should  provide  a 
mapping  between  data  processing  concepts  and  the  concept  within  an  operator’s 
domain  that  permits  functions  to  be  assessed  in  a  natural  and  intuitive  fashion.  Indeed, 
a  mental  model  is  a  representation  formed  by  the  operator  of  an  IAS,  based  on 
previous  experience  as  well  as  current  observation.  This  mental  model  provides  the 
operator  with  subsequent  system  understanding,  and  consequently  dictates  the  level  of 
task  performance  (Taylor,  1997); 

•  Training  in  the  use  of  these  IAS  is  necessary.  It  is  preferable  that  the  training  be 
embedded  within  the  system; 

•  The  IAS  should  monitor  interaction  for  errors  in  performance  and  suggest  a  corrective 
action,  which  should  be  based  upon  an  understanding  of  the  error  source  and  the 
probable  intent  of  the  operator  in  that  context; 

•  If  the  operator  is  to  regard  an  IAS  as  trustworthy  and  useful,  it  must  be  able  to  provide 
some  sort  of  explanation  for  its  current  or  intended  actions.  For  example,  it  may  be 
useful  to  provide  access  to  the  data  used  to  make  the  decision,  even  if  the  advice  was 
only  of  low  confidence;  and, 

•  Intelligent  Adaptive  Systems  should  be  designed  to  be  both  error-resistant  and  error- 
tolerant.  Error-resistant  systems  are  designed  to  prevent  operators  from  making  errors, 
whilst  error-tolerant  systems  are  able  to  recover  from  system  errors  in  a  safe  and 
timely  manner. 

5. 2. 2. 3  Human-System  Organisation 

Intelligent  adaptive  systems  share  responsibility,  authority  and  autonomy  over  many  system 
behaviours  with  the  human  operator.  Indeed,  the  motivation  for  creating  IASs  is  to  reduce 
operator  workload  and  information  overload.  However,  while  operators  wish  to  remain  in 


173 


charge  (and  it  is  desirable  for  them  to  do  so),  in  today’s  complex  systems,  operators  cannot  be 
fully  in  charge  of  all  systems  operations,  especially  not  in  the  same  way  they  have  been  in 
earlier  cockpits  and  workstations. 

Experience  with  these  systems  has  consistently  shown  that  this  concept  suffers  from  a  basic 
sociological  problem.  Namely,  the  human  operators  of  complex  systems  want  to  remain  in 
charge  of  the  equipment  they  use.  For  example,  the  Pilots  Associate  research  programme 
developed  a  list  of  prioritised  goals  for  a  good  cockpit  configuration  manager.  Two  of  the  top 
three  items  on  the  list  were  “Pilots  remain  in  charge  of  task  allocation”  and  “Pilots  remain  in 
charge  of  information  presented”  (Miller,  Pelican  and  Goldman,  1999). 

The  question  of  “who  is  in  charge”  was  addressed  by  Moss,  Reising  and  Hudson  (1984).  They 
suggested  that  task  and/or  decision  allocation  should  be  completed  according  to  the 
interaction  of  both  mission  goals  and  human/machine  capabilities.  Where  tasks  can  only  be 
performed  by  either  human  or  machine  their  allocation  is  simple.  Where  either  can  perform  a 
task,  then  the  task  should  be  allocated  by  consideration  of  “which  one  would  perform  the  task 
better?”  and  “what  will  be  the  mission  impact  of  such  an  allocation?”  They  suggest  that  the 
large  data  handling  capabilities  of  computers  suit  them  for  processing  the  large  amounts  of 
sensor  data  available.  This  is  achieved  through  knowledge  of  mission  goals  and  operator 
information  requirements,  so  that  the  data  can  be  collapsed  intelligently  into  a  form  readily 
interpreted  by  the  operator.  This  would  then  allow  the  operator  to  use  this  fused  data  to  make 
the  high-level  judgements,  decisions  under  uncertainty,  etc.,  at  which  humans  are  superior, 
thus  achieving  the  mission  goals  most  effectively.  Authority  allocation  will  be  an  essentially 
dynamic,  goal-oriented  process,  dependent  on  the  states  of  both  team  members  relative  to 
their  environment. 

However,  a  survey  conducted  by  NASA  (Tenney,  Rogers  and  Pew,  1995)  on  civil  pilot 
opinions  on  high  level  flight  deck  automation  issues  found  that  participants  were  nearly 
unanimous  in  proclaiming  that  the  pilot  will  still  be  responsible  for  flying  the  aircraft  in  the 
future.  They  also  showed  a  preference  for  automation  that  assists  the  pilot  in  problem  solving, 
as  compared  to  one  that  automatically  solves  problems.  The  majority  of  participants  felt  that 
the  biggest  needs  for  additional  automation  were  to  alleviate  further  mental  workload 
demands  imposed  on  them  in  time-constrained  decision  making  situations.  Overall,  pilots  in 
the  NASA  study  indicated  their  belief  for  the  pilot  to  remain  in  charge  so  that  the  automated 
systems  advise  rather  than  command. 

However,  a  more  recent  study  ascertained  the  opinions  of  Royal  Air  Force  aircrew  on 
automation  and  decision  support  that  would  improve  mission  effectiveness  and  SA,  as  well  as 
reducing  workload  (Banbury,  1997).  The  study  showed  significantly  different  opinions  as  to 
which  systems  should  have  automation  and  to  what  level.  Single-seat  aircrew  preferred 
increased  automation  of  the  defensive  sub-systems  to  the  point  where  little  human 
involvement  was  required.  This  suggests  that  the  single-seat  crew  had  less  cognitive  resources 
available  to  operate  the  defensive  systems  effectively  and  make  the  appropriate  defensive 
decisions.  The  general  consensus  of  opinion  was  in  favour  of  high  levels  of  automation, 
however,  aircrew  also  wanted  to  remain  in  the  decision  loop  and  have  an  interactive  role  in 
the  systems. 

A  study  on  aircrew  attitudes  towards  cockpit  automation  found  that  aircrew  wanted  high 
levels  of  automation  up  to  the  point  where  they  could  retain  ultimate  executive  control 
(Enterkin,  1994).  The  aircrew  reported  an  underlying  reluctance  to  place  complete  trust  in 
automated  systems  so  that  they  could  resume  control  if  the  system  is  in  error  or  non- 
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functional.  Opinions  relating  to  adaptive  automation  showed  that  aircrew  had  a  more 
favourable  opinion  towards  customised  automation  (i.e.,  operator-configured  automation)  and 
held  a  more  uncertain  view  of  intelligent  automation  (i.e.,  context-dependent  automation). 

There  were  a  number  of  reservations  voiced  about  customised  versus  intelligent  automation. 
Participants  felt  that  having  automation  levels  customised  to  the  individual  may  defeat  the 
objectives  of  having  standardised  equipment,  operating  procedures,  and  training  regimens.  In 
addition,  allowing  bespoke  cockpit  display  formats  and  configurations  may  negate  the  benefit 
of  standardised  training,  such  as  reduced  cost,  and  transfer  of  training  to  other  platforms. 
Attitudes  towards  having  automation  levels  specific  to  each  mission  phase  were  far  more 
positive,  because  this  aspect  of  customised  automation  can  be  readily  implemented  into 
training.  However,  the  report  did  not  solicit  any  opinion  from  pilots  as  to  what  level  of 
automation  would  be  desirable  for  each  mission  phase. 


Reference: 


Sheridan,  T.B.,  and  Parasuraman,  R.  (2006).  Human-automation  interaction.  In  R.S. Nickerson 
(Ed.).  Reviews  of  Human  Factors  and  Ergonomics,  Volume  1.  HFES:  Santa  Monica,  CA. 


Overview: 

This  paper  reviews  recent  research  in  the  area  of  human-automation  interaction.  It  describes 
taxonomies  including  supervisory  control  of  automation  and  function  allocation,  and  models  of 
human-automation  interaction.  The  paper  outlines  automation-related  accidents  associated  with 
inadequate  feedback  and  misuse  of  automation,  and  evaluates  the  social,  political,  and  ethical 
issues  related  to  role  of  etiquette  and  trust  on  operator  performance. 

Operator-automation/aqent  interaction:  The  authors  outline  three  ways  that  an  operator  can  interact 
with  a  system: 

1 .  By  specifying  to  the  automation/agent  the  task  goals  and  constraints  and  possible  trade-offs 
(e.g.,  pilots  programming  flight  management  systems); 

2.  By  controlling  the  automation/agent  to  start,  stop  or  modify  the  execution  of  the  automatic  task 
(e.g.,  clock  time;  abort  automatic  execution);  and, 

3.  By  receiving  information,  energy,  physical  objects,  or  substances  from  the  automation/agent, 
(e.g.,  warning  or  alarm  display;  expert  system  giving  advice). 

Supervisory  control  over  automated  systems:  A  new  relationship  between  the  operator  and  the 
system  is  identified,  whereby  the  operator  supervises  an  intelligent  but  subordinate  system  by 
issuing  instructions,  and  the  subordinate  executes  those  instructions  by  using  the  system’s  own 
memories,  built-in  programs,  sensors  and  energy  sources. 

Delegation  interfaces:  In  these  systems,  the  operator  delegates  tasks  to  the  system,  at  times  of  the 
operator’s  own  choosing  and  receives  feedback  on  their  performance. 


Conclusions  for  IASs: 

1 .  The  authors  outline  five  main  categories  of  techniques  for  implementing  adaptive  automation: 

•  Critical  events  method:  The  automation  is  triggered  by  critical  events  (e.g.,  when  pilot 
loses  consciousness,  the  auto-pilot  is  automatically  executed). 
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•  Operator  performance  and  physiological  measurements:  Automation  is  adapted  based 
on  an  assessment  of  operator  state.  For  instance,  this  could  include  using  a  secondary- 
task  measurement  technique  to  assess  operator  workload  in  a  primary  task  or  through 
Electroencephalographic  (EEG)  and  Event-Related  Potentials  (ERPs)  measurement. 

•  Modeling:  A  set  of  pre-defined  rules  for  implementing  adaptive  automation. 

•  Hybrid:  A  mix  of  the  other  four  methods. 

2.  The  authors  outline  various  advantages  and  disadvantages  of  the  five  methods: 

•  Critical  events  method  is  flexible  as  it  can  be  coupled  to  mission  planning,  but  does  not 
account  for  operator  requirements. 

•  Measurement  of  operator  performance  or  physiological  state  can  be  potentially 
responsive  to  unpredictable  changes  in  operator  cognitive  states.  Physiological 
measures  can  be  designed  to  be  relatively  unobstrusive,  and  have  high  bandwidth 
compared  with  performance  measures.  A  disadvantage  is  measurement  sensitivity, 
which  needs  to  be  established  in  each  application  domain. 

•  Modelling  techniques  can  be  implemented  offline  and  easily  incorporated  into  a  rule- 
based  expert  system.  However,  a  valid  model  is  required  and  different  models  within  the 
same  system  might  give  contrary  decisions  at  any  point  in  time. 

•  Hybrid  methods  attempt  to  optimize  relative  benefits  and  disadvantages  of  each 
technique,  and  may  therefore  offer  the  best  general  approach  to  implementing  adaptive 
automation. 

•  There  is  a  misconception  in  the  engineering  domain,  that  if  the  operator  is  removed  from 
the  system  (with  automation),  then  human  error  can  be  eliminated.  This  only  shifts  the 
focus  of  error  and  does  not  resolve  the  underlying  problem. 


Reference: 


Linegang,  M.P.,  Stoner,  H.A.,  Patterson,  M.J.,  Seppelt,  B.D.,  Hoffman,  J.D.,  Crittendon,  Z.B.,  and 
Lee,  J.D.  (2006).  Human-automation  collaboration  in  dynamic  mission  planning:  a  challenge 
requiring  an  ecological  approach.  Proceedings  of  the  50th  Human  Factors  and  Ergonomics  Society 
Conference,  San  Francisco,  CA. 


Overview: 

This  paper  presents  a  model  of  the  human-automation  interaction  system  and  examines  an 
ecological  approach  to  system  analysis  and  design.  According  to  the  authors,  this  model  provides  a 
theory  explaining  the  conflict  source  between  human  and  automation.  The  authors  claim  that  this 
model  predicts  that  an  ecological  approach  to  display  design  would  reduce  that  conflict. 

The  authors  advocate  a  goal-based  approach  to  adaptive  systems  based  on  a  control  theory 
framework.  The  human  specifies  goals  for  the  automated  system.  The  automation  system  then 
generates  a  set  of  planned  actions  and  executes  those  actions.  The  human  and  the  system  share 
the  role  of  implementing  automation.  That  is,  both  the  human  and  the  system  monitor  the 
environment  to  identify  “error”  that  would  necessitate  a  modification  of  the  plan. 
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The  authors  claim  that  conflicts  can  occur  in  this  shared  role  when  the  operator’s  implicit  goals 
(e.g.,  don’t  alert  an  enemy  to  your  presence),  do  not  match  the  pre-defined  goals  of  the  system 
(e.g.,  minimize  fuel  consumption).  To  minimize  this  conflict,  displays  should  represent  information 
relevant  to  explicit,  implicit,  and  pre-programmed  goals.  In  this  way,  the  operator  and  the  system 
can  collaborate  in  optimizing  the  achievement  of  all  true  mission  goals  (explicit  and  implicit).  The 
authors  recommend  an  “ecological”  approach  to  display  design  to  organize  information  according  to 
the  natural  structure  of  the  mission  environment. 

Mission  Displays  for  Autonomous  Systems  (MiDAS)  This  system  is  an  example  solution  to  resolve 
the  conflict  between  the  operator  and  the  system.  MiDAS  applies  an  ecological  approach  to  system 
design  and  involves  a  comprehensive  analysis  of  the  work  domain,  based  on  Work  Domain 
Analysis  (including  Cognitive  Work  Analysis)  and  Ecological  Interface  Design. 


Conclusions  for  IASs: 

1 .  The  authors  advocate  a  goal-based  approach  to  adaptive  systems  based  on  a  control  theory 
framework. 

2.  This  demonstration  provides  evidence  that  the  ecological  analysis  and  design  concepts  based 
on  this  analysis  can  transform  the  operator’s  display  into  a  tool  that  links  abstract  goals  to 
concrete  properties. 

3.  MiDAS  is  an  example  of  an  ecological  approach  which  can  allow  human  operators  to 
communicate  with  automation  systems  about  the  linkage  of  abstract  goals  with  concrete  plans 
and  properties  of  the  environment. 


Taylor,  R.  M.  (1998).  The  human-electronic  crew:  Human-computer  collaborative  team  working. 
Proceedings  of  the  1st  NATO  RTO  Human  Factors  and  Medical  Panel  Symposium  on  Collaborative 
Crew  Performance  in  Complex  Operational  Systems,  Edinburgh,  UK,  20-22  April  1998. 


Overview: 

This  paper  describes  the  concept  of  Human-Electronic  Crew  teamwork,  whereby  the  electronic 
crew  member  (the  electronic  support  system)  acts  as  an  associate  or  an  assistant,  sharing 
responsibility,  authority  and  autonomy  over  many  cockpit  tasks.  Various  methods  are  suggested  for 
ensuring  that  the  relationship  between  the  operator  and  the  system  is  flexible  and  adaptive 
including:  in-flight  situation  assessment  and  re-planning  (of  goals),  cognitive  modelling,  human 
intent  inferencing  and  error  recognition  (tracking  of  tasks),  and  the  use  of  complex  knowledge 
engineering  and  reasoning  logic  processes. 

Please  refer  to  Frameworks  Section  8.4.1  for  more  detailed  information  on  this  paper. 


Conclusions  for  IASs: 

1 .  The  authors  advocate  that  it  is  imperative  to  understand  the  operator’s  role  within  the 
system  to  determine  the  appropriate  system  support  for  that  role.  The  analysis  of  the 
operator’s  role  can  guide  the  system  design. 

_ 2.  Intelligent  aiding  systems  (e.g.,  full  associate  system)  should  provide  assistance  with 
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the  basic  functions  of  assessment,  planning,  co-ordinating  and  acting  (to  mimic  human 
information  processing  and  problem-solving  abilities). 

3.  Functional  architectures  are  a  good  way  to  implement  lAHs  that  support  strong 
interactions  and  tight  integration.  That  is,  the  behaviours  required  by  the  domains  (e.g., 
tasks)  are  shared  between  the  system  and  the  human  across  the  functional 
components. 

4.  In  order  to  support  an  associate  relationship  with  the  system,  the  authors  claim  that 
function  allocation  should  be  flexible  and  dynamic,  driven  more  by  the  situation  and 
context,  than  by  the  preservation  of  a  sole  sources  of  control  authority  (unlike  the 
CAMMY  and  CE  project  that  are  driven  by  pilot  control). 

5.  Operator  trust  is  enhanced  by  IAH  system  consistency  and  correctness  (e.g.,  decisions 
and  actions  are  consistent  and  predictable). 

6.  The  plan-goal  graph  (PGG)  modelling  approach  was  developed  to  address  the  problem 
of  intent  referencing  and  used  in  the  HEC  model  as  a  means  to  predict  pilot  intent. 
Intent  recognition  is  achieved  by  differentiating  the  goals  from  the  behaviour  of  the 
operator. 

7.  Operator  errors  with  increased  risk  of  severe  consequences  (especially  without 
corrective  action)  should  require  assertive  intervention  and  action  aiding  by  the  system 
(e.g.,  auto-pilot  is  automatically  turned  on  when  the  pilot  loses  consciousness). 

8.  The  system  should  conform  to  the  pilots’  mental  model.  A  mental  model  is  a 
representation  formed  by  an  operator  of  a  system  and/or  task  and  is  based  on  previous 
experience  and  current  observation.  This  provides  a  basis  for  the  operators 
understanding  of  system  functionality  which  can  influence  their  performance  on  tasks. 


Reference: 


Miller,  C.  and  Hannen,  M.  (1998).  User  acceptance  of  an  intelligent  user  interface:  A  rotorcraft 
pilot's  associate  example.  In  M.  T.  Maybury  (Ed.).  Proceedings  of  the  4th  International  conference 
on  intelligent  user  interfaces  (pp.  109-116).  New  York,  NY:  ACM  Press. 

Overview: 

This  paper  details  the  high  level  architecture  of  the  Cockpit  Information  Manager  (of  the  RPA).  It 
emphasizes  how  pilot  behaviours  are  monitored,  crew  intent  is  estimated,  symbols  are  selected  and 
de-cluttered,  windows  are  located,  and  the  automated  pan  and  zoom  and  the  allocation  of  tasks  are 
implemented. 

Please  refer  to  Frameworks  Section  8.4.2. 2  for  more  details  on  this  paper. 


Conclusions  for  IASs: 

1 .  A  full  associate  relationship  approach  is  taken  by  the  RPA  program  for  providing  system 
assistance. 

2.  This  approach  requires  that  pilot  intent  and  error  recognition  are  monitored  and  estimated  to 
provide  context  appropriate  assistance  (this  is  done  with  a  Crew  Intent  Estimator). 
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3.  A  World  State  or  “Context”  Model  is  used  to  determine  context. 

4.  A  task  and  goal-based  approach  in  the  form  of  a  Task  Network  is  used  to  estimate  pilot  intent 
and  error  recognition  with  pre-define  tasks/goals. 

5.  An  operator  profile  is  used  to  determine  pilot  expectations,  their  information  needs,  etc. 


6.  Preliminary  results  from  a  simulation  test  found  that  the  CIM  behaviours  are  contributing  to 
perceived  pilot  effectiveness,  reducing  workload  and  are  gaining  pilot  acceptance 


Geddes,  N.D.,  and  Shalin,  V.L.  (1997).  Intelligent  decision  aiding  for  aviation.  Technical  Report  (No. 
ISSN  1402-7585).  Prepared  for  Linkoping  Institute  of  Technology,  Linkpoping,  Sweden. 


Overview: 

Intelligent  decision  aiding  technologies  (adaptive  systems  and  automation)  are  reviewed  in  the 
context  of  the  aviation  domain.  This  report  explores  issues  of  system  architecture,  development  and 
integration  methods,  and  approaches  to  the  test  and  evaluation  of  large-scale  intelligent  aiding 
systems.  The  focus  is  based  on  the  Pilot’s  Associate  program.  The  following  briefly  describes  the 
development  strategies  used  to  develop  IAI  systems: 

•  Human-Centered  Design  (HMD)  perspective:  This  perspective  determined  that  the  role  of  the 
human  in  the  system  is  based  on  an  operational  philosophy.  It  identified  what  types  of  roles 
humans  should  play  in  the  system  (e.g.,  as  authority  and  agent). 

•  Design  Representations:  The  use  of  Intelligent  Object-Oriented  Design  (IOOD)  is  a  series  of 
steps  that  is  well  suited  to  transform  requirements  (e.g.,  system,  operator,  organization)  to  more 
abstract  views  of  objects  (refer  to  pages  65-68  for  details  on  this  process). 

•  Iterative  Design  Process:  This  process  recognizes  the  need  for  feedback  into  the  design  and 
requirements  process  to  ensure  that  the  design  of  the  system  evolves.  This  process  outlines  a 
series  of  prototyping  cycles  which  is  a  common  means  of  organizing  iterative  development. 

•  Application  of  development  tools:  It  was  found  that  an  iterative  development  is  most  productive 
when  supported  by  a  set  of  management,  design  and  testing  tools  (e.g.,  Plan  Goal  Graph  Tool, 
Display  Analyst). 


Conclusions  for  IASs: 

1 .  The  authors  found  that  knowledge-based  systems,  such  as  intelligent  adaptive 
systems,  require  more  elaboration  at  higher  levels  of  abstraction  (e.g.,  plan-goal 
graphs;  abstract  processes,  behaviours  and  use  case  templates).  This  can  be  achieved 
through  IOOD.  Object-oriented  languages  are  well  suited  to  transform  requirements 
and  support  the  development  of  IAH  systems. 

2.  Complex  intelligent  adaptive  systems  require  many  agents  and  frameworks  (i.e., 
operational  knowledge  representations).  Interaction  protocols  can  be  used  to  ensure 
that  the  system  operates  as  a  whole. 

3.  There  are  now  a  large  number  of  well-developed  reasoning  algorithms  and  operational 
knowledge  representations.  While  a  wealth  of  processes  (i.e.,  algorithms)  and 

_ representations  is  often  necessary  for  complex  systems,  the  authors  are  careful  to 
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indicate  that  the  system  should  take  the  form  that  best  suits  the  process  desired. 

4.  The  authors  have  noted  that  the  development  process  requires  tool  support  (e.g., 
PGG)  but  that  the  tools  presently  available  (as  of  1997)  fall  short  for  knowledge-based 
systems.  System  designers  must  therefore  anticipate  the  need  to  develop  tools  along 
with  the  product. 


Mooshage,  O.  Distelmaier,  H  &  Grandt,  M.  (2002).  Human  Centered  Decision  Support  for  Anti-Air 
Warfare  on  Naval  Platforms.  In  Proceedings  of  RTO  Human  Factors  and  Medicine  Panel  (HFM) 
Symposium  held  in  Warsaw,  Poland,  7-9  October  2002. 


Overview: 

This  paper  examines  various  methods  to  acquire,  describe,  formalize,  and  represent  knowledge 
and  problem  solving  strategies  of  domain  experts  for  the  development  of  rule-based  intelligent 
decision  aids  in  Anti-Air  Warfare  (AWW). 


Conclusions  for  IASs: 

1 .  An  intelligent  decision  aid  should  support  the  resources  specific  to  all  stages  (roles)  of  human 
information  processing  (i.e.,  based  on  Wickens’  model  of  multiple  resources). 

2.  The  authors  recommend  that  a  system  should  provide  support  according  to  the  complexity  of 
the  task  at  hand,  the  over  all  situation  and  the  operator  state. 

3.  The  authors  recommend  that  knowledge  representation  techniques  should  enable  the  operator 
to  retain  an  adequate  situational  awareness.  They  found  that  certain  knowledge-based 
techniques  can  provide  information  about  how  a  system  determined  a  conclusion,  while  other 
systems  cannot.  For  this  reason,  neural  networks  are  not  considered  a  good  strategy  as  they 
lack  any  transparency.  Generic  algorithms  are  also  not  appropriate  due  to  their  inherent 
mutation  that  makes  them  unpredictable. 

4.  The  authors  recommend  that  formal  decision  support  techniques  beyond  rule-based  support 
systems  should  be  used  to  support  the  human  decision  maker’s  cognitive  processes  (i.e.,  level 
of  information  processing).  Three  techniques  are  recommended: 

•  Bayesian  belief  networks  (suitable  for  problems  for  which  a  substantial,  representative 
collection  of  successfully  solved  cases  is  available); 

•  Case-based  reasoning  (for  classification  problems  for  which  enfolded  collections  of  real 
or  made  up  cases  with  correct  solutions,  and  detailed  logging  of  all  attributes  exist); 

•  Fuzzy  theory  (can  be  applied  to  a  task  in  which  the  degree  of  conformance  can  neither 
be  set  assuredly  “true”  or  “false”  but  merely  mapped  to  fuzzy  sets). 

5.  Variables  related  to  decision  making  should  be  identified  using  empirical  knowledge  acquisition 
techniques  (e.g.,  laboratory  test  bed). 

6.  A  purely  rule-based  support  system  is  not  capable  of  supporting  the  human  decision  maker  in  a 
situation  in  which  knowledge-based  acting  is  essential. 

7.  While  certain  knowledge-based  techniques  can  provide  auxiliary  information  about  the  way  that 
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led  to  their  conclusions,  others  cannot.  For  this  reason,  neural  networks  are  not  considered  a 
good  strategy  as  they  lack  any  transparency.  Generic  algorithms  are  also  not  appropriate  due 
to  their  inherent  mutation  that  makes  them  unpredictable. 


Reference: 


Miller,  C.,  Funk,  H.,  Wu,  P.,  Goldman,  R.,  Meisner,  J.,  Chapman,  M.  (2005).  The  Playbook 
Approach  to  Adaptive  Automation,  In  Proceedings  of  the  Human  Factors  and  Ergonomics  Society's 
49th  Annual  Meeting,  Orlando,  FL 


Overview: 

Playbook  is  a  human-system  communication  tool  that  allows  delegated  control  of  automation.  The 
tool  is  based  on  a  shared  model  of  the  tasks  in  a  domain.  This  shared  task  model  provides  a  means 
of  human-automation  communication  about  plans,  goals,  methods  and  resource  usage,  a  process 
similar  to  referencing  plays  in  a  sports  team’s  playbook.  The  Playbook  enables  operators  to  interact 
with  subordinate  systems  with  human  subordinates,  thus  allowing  for  adaptive  automation.  This 
approach  and  its  application  is  described  through  an  ongoing  project  called  Playbook-enhanced 
Variable  Autonomy  Control  System™ 

Playbook  is  a  specific  method  of  implementing  a  delegation  interaction  which  can  be  divided  into 
two  components:  (1)  a  hierarchical  task  model  that  is  compatible  with  levels  of  automation  (cf. 
Sheridan,  1987);  and  (2)  a  planning  mechanism  for  evaluating  existing  resources,  plan  validity,  and 
instantiating  the  task  models. 

A  shared  task  model  is  comprised  of  a  set  of  play  templates  are  generated  by  identifying  a  set  of 
common  tasks,  grouping  those  tasks  into  plays,  and  enabling  elements  such  as  time  and  location  to 
become  task  parameters. 

How  Playbook  works.  When  a  previously  defined  play  is  executed,  the  operator  can  select  a  play 
template  and  apply  the  parameter  values  as  appropriate  to  his/her  needs.  Both  the  operator  and  the 
automation  have  a  similar  model  of  the  sequence  of  tasks  to  execute  (the  shared  task  model). 

The  overall  Playbook  architecture  consists  of  three  components:  a  library  of  task  models;  a 
constraint-based  planning  engine;  and  an  OMI. 


Conclusions  for  IASs: 

1 .  Findings  provide  support  for  allowing  the  tasking  of  multiple  agents  while  keeping  the 
supervisor  in  the  decision-making  loop,  without  increasing  supervisor  mental  workload.  It 
also  suggests  that  the  human  supervisor  can  adapt  successfully  to  unpredictable  changes 
in  the  environment. 

2.  Playbook  provides  a  complete  architecture  for  the  integration  of  human  input,  intelligent  a 
priori  planning,  reactive  planning  and  event  handling,  and  ongoing  vehicle  control  loops. 

3.  The  authors  recognize  that  new  methodologies  are  still  needed  to  build  more  extensive 
task  models.  For  instance,  Playbook  task  knowledge  should  arise  from  results  of  Cognitive 
Work  Analysis  of  a  task  domain  and  then  the  Playbook  architecture  (including  Ul  and 
planning  components)  can  be  used  to  produce  useful  task  timeline  inputs  for  a  constructive 
simulation 
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Reference: 


Onken,  R.  (2002).  Cognitive  Cooperation  for  the  Sake  of  the  Human-Machine  Team  Effectiveness. 
In  Proceedings  of  RTO  Human  Factors  and  Medicine  Panel  (HFM)  Symposium  held  in  Warsaw, 
Poland,  7-9  October  2002. 


Overview: 

This  paper  is  a  keynote  address  presenting  a  framework  for  incorporating  cognitive  automation  into 
a  work  system.  A  team  coordination  approach  is  suggested. 

Cognitive  automation  is  based  on  the  comprehensive  knowledge  about  work  process  objectives 
and  goals  on  all  goal  hierarchy  levels.  Task  options  and  necessary  data  describing  the  current 
situation  in  the  work  process  are  also  reviewed.  Cognitive  automation  is  a  prime-goal-oriented 
based  approach. 


Conclusions  for  IASs: 

1 .  The  authors  advocate  that  the  coordination  of  authority  for  task  initiation  automation  requires 
that  cooperating  team  members  should  know  as  much  as  possible  from  each  other’s 
performance  characteristics  and  behavioural  traits.  Therefore,  modelling  of  cognitive  behaviour, 
in  particular  of  human  behaviour,  is  needed  to  enhance  machine-human  team  effectiveness 

2.  The  authors  report  that  the  success  of  cognitive  cooperation  is  highly  dependent  on  a  valid 
model  of  the  human  operator.  The  validity  of  this  model  will  influence  the  capability  of  the 
artificial  cognitive  units  to  provide  appropriate  aid. 


5.2. 2.4  Human-System  Teamwork 

Another  important  research  area  that  should  be  considered  for  IASs  is  theories  of  teamwork. 
A  number  of  the  IASs  discussed  previously  are  able  to  learn  through  experience,  and  to 
anticipate  the  actions  of  the  operator  based  on  previous  behaviour  and  mission  objectives. 
Potentially,  this  increases  the  functional  effectiveness  of  such  a  human-machine  team.  This 
concept  of  human  and  machine  working  as  an  intelligent,  co-operative  team  is  considered  by 
many  as  being  central  to  the  successful  application  of  IAS  (e.g.,  Emerson,  Reinecke,  Reising 
and  Taylor,  1988;  1990). 

A  model  of  teamwork  is  described  by  Taylor  and  Selcon  (1993).  Their  model  is  derived  from 
the  social  psychology  of  small  group  dynamics  (Figure  8).  Teams  are  considered  to  differ 
from  small  groups  in  that  greater  emphasis  is  placed  on  clear  definition  of  goals,  roles  and 
structure  for  teams.  Taylor  and  Selcon  claim  that  teams  have  three  distinctive  characteristics: 

1 .  Co-ordination  of  activity,  aimed  at  performing  certain  tasks  and  at  achieving  specific, 
agreed  goals.  Such  co-ordination  is  dependent  on  trust  between  team  members  to  be 
successful,  since  trust  is  the  mechanism  which  allows  co-ordination  of  effort  to  take 
place; 

2.  Communication  and  interaction  between  team  members;  and, 


182 


3.  Well-defined  organisation  and  structure,  with  members  occupying  specific  roles  with 
associated  power,  authority  and  status,  whilst  exhibiting  conformity  and  commitment 
to  team  norms  and  goals.  Such  organisation  will  define  the  allocation  of  functions  and 
the  locus  of  authority  within  the  team. 


Figure  8:  Model  of  Human-Electronic  Teamwork  (Taylor  and  Selcon,  1990, 1993). 


The  system  of  relationships  between  the  components  of  teamwork  can  be  understood  in  terms 
of  the  team’s  goals,  resources,  and  their  effects  on  individual  team  members,  team 
development  and  team  performance.  Such  a  model  provides  the  framework  for  considering 
the  implementation  of  IASs  so  as  to  produce  an  effective  team  capable  of  best  achieving  the 
operational  aims  for  which  it  is  designed. 

Implementing  IASs  in  a  way  that  produces  a  synergistic  relationship  with  the  human  operator 
raises  a  number  of  human-computer  interaction  issues,  the  solutions  which  are  likely  to  be 
crucial  in  any  successful  system.  Foremost,  among  these  must  be  considerations  of  where 
authority  should  be  vested  within  the  team  and  human  trust  in  the  system. 


Reference: 


Onken,  R.  (2002).  Cognitive  Cooperation  for  the  Sake  of  the  Human-Machine  Team  Effectiveness. 
In  Proceedings  of  RTO  Human  Factors  and  Medicine  Panel  (HFM)  Symposium  held  in  Warsaw, 
Poland,  7-9  October  2002. 
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Overview: 


This  paper  is  a  keynote  address  presenting  a  framework  for  incorporating  cognitive  automation  into 
a  work  system.  A  team  coordination  approach  is  suggested. 

Cognitive  automation  is  based  on  the  comprehensive  knowledge  about  work  process  objectives 
and  goals  on  all  goal  hierarchy  levels.  Task  options  and  necessary  data  describing  the  current 
situation  in  the  work  process  are  also  reviewed.  Cognitive  automation  is  a  prime-goal-oriented 
based  approach. 


Conclusions  for  IASs: 

1 .  The  authors  advocate  that  the  coordination  of  authority  for  the  initiation  requires  that 
cooperating  team  members  should  know  as  much  as  possible  from  each  other’s  performance 
characteristics  and  behavioural  traits.  Therefore,  modelling  of  cognitive  behaviour,  in  particular 
of  human  behaviour,  is  needed  to  enhance  machine-human  team  effectiveness 

2.  The  authors  report  that  the  success  of  cognitive  cooperation  is  highly  dependent  on  a  valid 
model  of  the  human  operator.  The  validity  of  this  model  will  influence  the  capability  of  the 
artificial  cognitive  units  to  provide  appropriate  aid. 


Reference: 


de  Reus,  A.J.C.,  Roessingh,  J.J.M.,  and  Pouw,  F.C.M.  (2006).  Modelling  Approach  to  Teamwork 
Issues  in  a  UAV  Ground  Control  Station.  In  Proceedings  of  RTO  Human  Factors  and  Medicine 
Panel  (HFM)  Symposium  held  in. 


Overview: 

This  paper  discusses  a  high-level  qualitative  model  for  teamwork  as  a  means  to  capture  the 
teamwork  processes.  This  model  in  turn  provides  the  basis  for  lower  level  Bayesian  Belief  Network 
models  that  focus  on  mishap  scenarios. 

Model  development  process: 

First,  a  team  task  analysis  was  undertaken,  which  resulted  in  the  identification  of  skills  for 
teamwork  in  UAV  operations.  Secondly,  using  the  high  level  model,  different  lower  level  models 
were  defined  in  the  form  of  a  Bayesian  Belief  Networks.  The  goal  of  the  modelling  effort,  taking  the 
two-step  approach,  is  to  estimate  the  relative  contribution  of  team  skills  in  UAV  mishaps,  with  focus 
on  such  skills  as  team  resource  management  skills,  team  decision-making,  insight  in  the 
automation  concept,  and  (shared)  knowledge  of  standard  operating  procedures,  including  selection 
of  the  appropriate  procedures.  The  Bayesian  Belief  Networks  was  found  to  be  a  useful  tool  to 
structure  task  allocation,  task  sharing,  team  composition,  procedures,  and  OMIs  from  a  teamwork 
perspective. 


Conclusions  for  IASs: 

1 .  Teamwork  concepts  should  be  considered  when  designing  systems  that  require  human 
operators  to  simultaneously  control  several  functions  at  various  working  positions,  such  as 
controlling  multiple  UAVs. 

2.  This  paper  demonstrated  that  Bayesian  Belief  Networks  can  be  used  to  structure  task 
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allocation,  task  sharing,  team  composition,  procedures,  and  OMIs  from  a  teamwork 
perspective. 


Franklin,  D.,  Budzik,  J.,  and  Hammond,  K.  (2002).  Plan-based  Interfaces:  Keeping  Track  of  User 
Tasks  and  Acting  to  Cooperate.  In  IUI’02,  January  13-16,  2002. 


Overview: 

This  paper  describes  the  concept  of  an  Intelligent  Classroom,  which  consists  of  a  computer  system 
that  dynamically  adapts  to  operator  actions  and  inputs  (gesture  and  voice)  in  a  classroom 
environment  (i.e.,  controls  camera,  automatic  presentation  slide-switcher).  The  algorithms  are 
goal-based  and  driven  by  task  recognition. 

Intelligent  Classroom:  The  1C  is  an  automated  lecture  facility  prototype  that  serves  as  its  own 
audio/visual  assistant.  The  operator  (e.g.,  speaker),  provides  a  presentation,  and  the  Classroom 
watches  and  listens,  and  when  appropriate,  assists  will  provide  assistance.  The  1C  keeps  track  of 
various  activities  pursued  by  the  speaker  as  well  as  its  own  activities  in  control  of  its  various 
autonomous  components. 

The  representation  is  used  three  ways  to  accomplish  a  goal: 
plan  execution  (execute  a  plan  to  achieve  a  goal),  plan 
recognition  (match  the  operator’s  actions  to  a  set  of  known 
plans),  and  projection  (follows  the  operator’s  plan  and  projects 
future  actions).  A  set  of  agents  are  used  to  monitor,  recognize, 
and  execute  some  plan  to  accomplish  an  operator  goal. 

The  system  is  based  on  the  principle  that  the  world  is 
composed  of  a  series  of  processes.  A  process  is  a  single 
agent  that  executes  a  sequence  of  actions.  It  is  composed  of  one  or  more  discrete  steps,  each  of 
which  specifies  a  number  of  continuous  actions  and  a  number  of  discrete  events.  The  processes 
are  designed  such  that  the  Classroom  can  essentially  use  the  same  algorithm  for  executing  a 
process  that  it  used  for  observing  the  operator  as  the  operator  executes  a  process. 

To  alter  the  algorithm  so  that  the  Classroom  can  observe  the  operator  and  to  follow  along  with  the 
operator’s  plans,  only  a  portion  of  the  first  step  needs  to  be  changed.  Rather  than  performing  the 
primitive  actions  that  are  a  part  of  the  step,  the  Classroom  performs  “observation”  actions  that 
complement  the  primitive  actions. 

The  Process  manager  continually  steps  through  its  set  of  processes  to  keep  them  synchronized 
with  the  operator  and  revises  the  set  of  processes  when  required. 

Human-machine  cooperation.  The  operator,  in  executing  part  of  a  plan,  expects  the  Classroom  to 
do  its  part  of  the  plan.  A  plan  is  a  set  of  processes  (often  to  be  executed  by  a  number  of  different 
agents)  that  when  executed  together  successfully,  accomplish  some  goal.  In  the  Classroom,  most 
plans  have  one  process  executed  by  the  operator  and  one  or  two  processes  executed  by  the 
Classroom.  This  definition  makes  explicit  the  presence  of  other  agents  or  exogenous  events.  In  the 
Classroom,  these  plans  attempt  to  express  a  common  understanding  of  how  a  speaker  and  an 
audio/visual  assistant  interact. 
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Conclusions: 


1 .  The  same  techniques  implemented  in  the  Intelligent  Classroom  can  be  applied  to  a  broad 
range  of  interactive  applications.  Refer  to  the  paper  for  details  on  how  to  implement 
techniques. 

2.  The  system  should  understand  the  operator’s  actions  in  the  context  of  what  it  believes  the 
operator  is  doing. 

3.  The  ability  to  provide  reason  to  the  operator’s  activity  is  crucial  to  the  implementation  of  an 
intelligent  operator  interface. 

4.  Plan  generation  and  recognition  are  a  promising  means  of  adaptive  automation  and  estimating 
pilot  intent. 

5.  Human-machine  cooperation  can  be  achieved  by  allowing  an  operator,  when  executing  a  part 
of  a  plan,  to  expect  a  system  to  help  in  executing  that  part  of  the  plan.  A  plan  is  a  set  of 
processes  (often  to  be  executed  by  a  number  of  different  agents)  that  when  executed  together 
successfully,  accomplish  some  goal.  Plans  attempt  to  express  a  common  understanding  of 
how  a  speaker  and  an  audio/visual  assistant  interact. 


Reference: 


Wolfman,  S.A.,  Lau  Pedro  Domingos,  T  and  Weld,  D.S.  (2001).  Mixed  Initiative  Interfaces  for 
Learning  Tasks:  SMARTedit  Talks  Back.  In  proceedings  of  IUI’01,  January  14-17,  2001,  Santa  Fe, 
New  Mexico,  USA. 


Overview: 

An  interface  for  machine  learning  is  proposed.  The  paper  describes  a  variety  of  interaction  modes 
that  enhance  the  learning  process  and  presents  a  decision-theoretic  framework,  called  DIAManD, 
for  choosing  the  best  interaction. 

The  authors  propose  that  machine  learning  systems  should  closely  resemble  human  teacher- 
student  relationships  and  follow  the  example  of  the  proactive  yet  considerate  student.  For  instance, 
the  system  should  ask  questions,  propose  examples  and  solutions,  and  relate  its  level  of 
knowledge  when  appropriate  to  make  the  interaction  more  effective. 

DIAManD  is  a  system  for  selecting  among  various  interaction  modes  using  a  multi-attribute  utility 
function.  The  interaction  modes  provide  a  variety  of  methods  for  an  operator  to  interact  with  the 
system.  The  system  selects  from  a  set  of  interaction  modes  the  mode  it  judges  most  appropriate 
based  on  attribute  vectors.  The  best  of  these  modes  is  presented  to  the  operator  and  controls  the 
next  stage  of  discourse,  updating  the  state  of  the  learner.  The  modes  are  then  rescored  based  on 
the  new  state  of  the  learner. 
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Conclusions  for  IASs: 

1 .  The  authors  advocate  a  mixed-initiative  interface  in  which  the  machine  learner  and  human 
operator  equally  share  responsibility  for  guiding  the  learning  process. 

2.  A  learning  system  should  have  several  modes  of  interaction  with  the  operator  to  acquire 
the  concepts  more  quickly  (e.g.,  through  judicious  choice  of  the  example  to  classify,  as  in 
active  learning)  and  should  allow  the  operator  to  have  more  control  over  the  learning 
process.  See  paper  for  details  on  interaction  modes. 

3.  A  mixed-initiative  framework  (e.g.,  DIAManD),  where  the  learner  and  human  operator  are 
each  participants  in  a  dialogue,  could  improve  the  learner's  hypothesis  with  minimal  effort 
on  the  part  of  the  operator. 

4.  To  facilitate  rapid  learning,  the  interface  should  provide  some  mechanism  for  feedback  to 
the  learning  system  on  particularly  poor  interaction  mode  choices  (the  feedback  model  is 
further  described  in  the  article). 


Reference: 


Eggleston,  R.G.,  Roth,  E.M.,  and  Scott,  R.A.  (2003).  A  Framework  for  Work-Centered  Product 
Evaluation.  In  proceedings  of  the  47th  Human  Factors  and  Ergonomics  Society  Conference, 
Denver. 


Overview: 

A  comprehensive  work-centred  evaluation  framework  that  assesses  new  technology  for  their  value 
in  supporting  human  performance  is  described.  A  key  feature  of  the  framework  is  that  it 
encompasses:  usability,  usefulness  and  impact.  This  concept  is  illustrated  through  a  work-centred 
support  system  prototype.  The  framework  is  detailed  in  Young,  M.J.  and  Eggleston,  R.G.  (2002). 

In  the  WCSS  prototype,  software  agents  are  designed  as  small,  independent  chunks  of  software 
that  address  tasks  as  separately  controlled  and  modifiable  modules.  This  enables  software 
components  to  be  organized  according  to  functional  elements  of  work  in  a  particular  domain. 

A  detailed  domain  analysis  was  performed  to  map  domain  work  requirements  and  systematically 
allocate  tasks  to  human  and  software  agents. 

Structuring  agents  in  functional  terms  provides  a  concrete  vocabulary  of  concepts  and  metaphors 
that  can  be  shared  among  software  engineers,  cognitive  engineers,  and  operators. 

Two  types  of  interfacing  agents  are  used  in  the  prototype.  The  “visibility”  of  the  agents  is  based  on 
the  task: 

•  Agents  organized  around  domain  work.  These  include  forecasting  agents,  region  analysis  and 
mission  analysis  agents,  which  are  agents  that  operators  “delegate”  work  to;  they  have  no 
personality. 

•  Agents  that  operators  can  access  if  needed.  These  include  data  acquisition  agents. 
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Conclusions  for  IASs: 

1 .  Cognitive  work  analysis  is  an  effective  means  of  establishing  system  requirements.  The 
authors  advocate  that  sources  of  cognitive  and  collaborative  demands  should  be  analyzed  in 
the  applied  domain  and  should  involve  close  interaction  among  the  cognitive  engineers, 
software  developers  and  domain  practitioners. 

2.  Automated  agents  should  act  as  ‘team  players’. 

3.  Visibility  of  agents:  Automated  agents  need  to  be  observable  (or  transparent/visible)  so  that 
operators  are  able  to  determine  the  current  state  of  the  automated  agents,  and  understand 
what  the  agents  will  do  next  relative  to  the  state  of  the  task.  The  amount  of  “visibility”  required 
is  questionable  (i.e.,  the  issue  of  trust  and  mistrust  can  occur  or  fully  visible  such  as  the 
Microsoft  “PaperClip”  which  takes  advantage  of  assistant  and  subordinate  metaphors) 

4.  Humans  should  have  control  and  be  able  to  re-direct  the  software  agents  as  task 
requirements  change. 

5.  A  system  needs  to  support  multiple  facets  of  individual  cognitive  and  collaborative  work.  This 
involves  consideration  of  problem-solving/decision-making  aspects  of  work,  activities  involved 
in  creating  work  products,  processes  involved  in  collaborative  work,  and  the  cognitive  effort 
involved  in  tracking  and  managing  multiple  intertwined  work  activities. 

6.  Object-oriented  design  techniques  are  useful  in  facilitating  collaboration  between  operators, 
cognitive  engineers,  and  software  engineers  (although  as  system  complexity  increases,  the 
operator  can  lose  sight  of  the  big  picture). 

7.  Agent-based  architectures  provide  potential  for  operator-accessible  descriptions  of  domain 
objects,  workflow,  and  large-scale  interactions  between  domain  objects. 


Reference: 


Sheridan,  T.B.,  and  Parasuraman,  R.  (2006).  Human-automation  interaction.  In  R.S. Nickerson 
(Ed.).  Reviews  of  Human  Factors  and  Ergonomics,  Volume  1.  HFES:  Santa  Monica,  CA. 


Overview: 

This  paper  reviews  recent  research  in  the  area  of  human-automation  interaction.  It  describes 
taxonomies  including  supervisory  control  of  automation  and  function  allocation,  and  models  of 
human-automation  interaction.  The  paper  outlines  automation-related  accidents  associated  with 
inadequate  feedback  and  misuse  of  automation,  and  evaluates  the  social,  political,  and  ethical 
issues  related  to  role  of  etiquette  and  trust  on  operator  performance. 


Conclusions  for  IASs: 

1 .  Levels  of  automation  (assigned  agent:  operator  or  system)  and  initiation  of  task.:  The  authors 
recognize  that  there  are  various  degrees  of  automation  appropriate  to  different  contexts,  and 
that  different  process  stages  (i.e.,  information  acquisition,  analysis,  decision  about  action, 
implementation  of  action)  of  a  complex  system  are  appropriately  automated  to  different 
degrees. 

2.  Proper  feedback  and  communication.  Systems  should  provide  the  operator  with  information 
concerning  automation  modes,  system  states,  and  future  automated  actions.  This  may 
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improve  human-machine  communication  and  therefore  potentially  enhance  system 
performance. 

3.  Ecological  Interface  Design  displays  for  non-routine  conditions.  EID  displays  may  be 
particularly  helpful  for  operator-automation  interaction  under  non-routine  conditions.  When 
unexpected  events  occur,  an  operator-automation  display  interface  is  required  that  allows  for 
quick  comprehension  of  system  state  and  rapid  action. 

4.  The  authors  note  that  one  of  the  main  problems  of  operator-system  automation  is  not  one  of 
authority  or  autonomy  but  of  cooperation  and  observability.  Cooperation  means  a  shared 
mental  representation  between  the  operator  and  the  system. 

5.  The  authors  identify  the  “mixed-initiative  problem”;  as  systems  become  more  complex,  more 
and  more  control  loops  are  required  and  the  probability  of  these  control  loops  interfering  with 
each  other  increases.  This  can  occur  if  the  operator  cannot  see  what  actions  the  other  control 
loops  are  performing  or  what  loops  are  being  accessed  from  the  same  resource  pool. 


5.2.2.5  Supervisory  Control  over  Initiation  of  Adaptation 

Supervisory  control  over  automated  systems  is  a  form  of  human-machine  relationship;  the 
human  supervises  an  intelligent  but  subordinate  system  by  issuing  instructions,  and  the 
subordinate  executes  those  instructions  by  using  their  own  memories,  built-in  programs, 
sensors  and  energy  sources.  These  systems  are  also  known  as  delegation  interfaces;  operators 
delegate  tasks  to  the  system,  at  times  the  operator  chooses,  and  receives  feedback  on  their 
performance. 


Miller,  C.  and  Goldman,  R.  (1999).  Tasking  interfaces:  Associates  that  know  who's  the  boss.  In  J. 
Reising,  R.  M.  Taylor  and  R.  Onken  (Eds.).  The  human  electronic  crew:  The  right  stuff? 
Proceedings  of  the  4th  joint  GAF/RAF/USAF  workshop  on  human-computer  teamwork,  Kreuth, 
Germany  (Technical  report  AFRL-HE-WP-TR-1 999-0235  pp. 97-102).  Wright  Patterson  AFB,  OH: 
Air  Force  Research  Laboratory. 


Overview: 

This  paper  describes  the  techniques,  adapted  from  the  “associate”  (PA)  research,  used  for  the 
construction  of  tasking  interfaces.  They  present  initial  work  on  a  solution,  which  allows  human 
operators  to  interact  with  advanced  automation  at  various  levels.  According  to  this  model,  tasked 
systems  should  always  be  sub-ordinate,  but  must  know  enough  about  the  tasks  in  the  domain.  The 
authors  claim  that  instructing  these  “tasking  interfaces”  is  vastly  easier  than  instructing  traditional 
automated  systems.  Concepts  are  described  and  discussed  in  the  context  of  a  tasking  interface  for 
UAVs. 

Playbook  OMI 

This  is  an  interface  that  allows  the  operator  to  inspect  and  interact  with  the  system  (through  a  task 
model)  by  “calling  plays”  and  activating  tasks  at  various  levels  and  sub-levels.  Through  this _ 
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interface,  the  operator  will  graphically  instruct  a  full  or  partial  plan  for  the  mission  by  specifying  the 
tasks  to  be  performed,  or  goals  to  be  accomplished  by  the  system  (Figure  6). 

Plavbook  Framework: 

The  framework  is  composed  of  four  primary  components: 

•  Playbook  OMI 

•  Mission  Analysis.  A  projective  planning  system  which  is  capable  of  understanding 
instructions  from  the  operator  through  the  OMI. 

•  Event  Handling.  All  accepted  plans  are  passed  from  the  mission  analysis  module  to  “even 
handling”  where  plans  can  be  adjusted  in  real-time. 

•  Control  algorithms.  Executes  the  instructions. 

This  framework  is  based  upon  and  interacts  with  a  Shared  Task  Model  Infrastructure,  which  can 
facilitate  human-system  coordination. 


Conclusions  for  IASs: 

1 .  The  authors  stress  that  a  usability  evaluation  of  the  tasking  interface  GUI  (and  all  system 
interfaces)  is  required. 

2.  The  authors  warn  that  tasking  interfaces  should  not  rely  on  a  predefined  set  of  task 
models,  but  dynamic  ones.  The  operator  should  be  able  to  create  novel  tasks  and  to  store 
components  of  models  which  are  useful. 

3.  The  authors  acknowledge  that  their  task  network  representation  is  weak  in  its  coding  of 
goals,  which  are  seen  as  a  critical  component  of  any  tasking  interface. 

4.  Operators  need  sufficient  training  for  interacting  with  the  tasking  interface. 

5.  A  delegated  interface  may  increase  operator  acceptance;  that  is,  by  enabling  a  system  to 
behave  more  like  an  intelligent  subordinate,  operators  may  be  more  tolerant  of  their 
weaknesses  and  more  acceptable  of  their  capabilities  in  a  controlled  setting. 


Parasuraman,  R.,  and  Miller,  C.  (2006).  Delegation  interfaces  for  human  supervision  of  multiple 
unmanned  vehicles:  Theory,  experiments  and  practical  applications.  Advances  in  Human 
Performance  and  Cognitive  Engineering  Research,  7,  251-266 


Overview: 

This  paper  provides  a  framework  for  human  supervision  of  multiple  UAVs  based  on  the  concept  of 
delegation.  Delegation  type  interfaces  represent  a  form  of  adaptive  or  adaptable  automation.  This 
paper  claims  that  delegation  interfaces  can  retain  the  benefits  of  automation,  while  minimising 
some  costs.  Results  from  laboratory  experiments  are  presented  to  illustrate  the  influence  of 
delegated  interfaces  on  operator  performance  and  acceptance. 


190 


Conclusions  for  IASs: 

1 .  Results  show  that  these  interfaces  allow  operators  to  respond  effectively  to  unpredictable 
changes,  which  was  associated  with  the  flexibility  afforded  by  delegation  interfaces. 

2.  Slightly  higher  workload  was  found  when  participants  had  flexible  access  to  both  manual 
control  and  automated  plays. 

3.  Flexible  interfaces  were  preferred  over  full  or  no  automation. 

4.  Findings  suggest  that  while  flexible  interfaces  are  preferred,  workload  can  still  be  increased. 
Therefore,  a  balance  between  operator  acceptance  and  workload  should  be  maintained. 


Cummings,  M.L.  (2004).  The  need  for  command  and  control  instant  message  adaptive  interfaces: 
Lessons  learned  from  tactical  Tomahawk  human-in-the-loop  simulation.  CyberPsychology  and 
Behaviour,  7(6),  653-661 


Overview: 

This  paper  examines  human  performance  issues  for  supervisory  control  of  the  Navy's  new  Tactical 
Tomahawk  missile.  Measurements  of  operator  situation  awareness  and  workload  through  a 
secondary  tasking  were  taken  through  an  embedded  instant  messaging  program.  Discussed  are 
the  first  attempts  to  quantify  human  supervisory  command  and  control  performance  degradation  as 
a  result  of  interference  from  an  instant  messaging  secondary  tasking. 

In  the  course  of  human-in-the-loop  experiments,  two  performance  measures  were  taken  through 
the  instant  messaging  interface:  1)  secondary  tasking  as  a  measure  of  workload;  and  2)  situation 
awareness.  Performing  a  secondary  tasking  is  a  commonly  used  workload  measurement  tool  that 
requires  a  subject,  assigned  to  a  primary  task,  to  use  any  spare  mental  capacity  to  attend  to  a 
secondary  task. 

Results  revealed  that  some  subjects  fixated  on  the  real-time  instant  messaging  secondary  task 
instead  of  the  primary  task  of  missile  control,  leading  to  the  overall  degradation  of  mission 
performance  as  well  as  a  loss  of  SA.  To  effectively  address  the  interruption  problem  in  instant 
messaging,  one  challenge  was  to  develop  an  adaptive  system  in  which  the  computer  schedules 
interruptions  to  occur  in  periods  of  low  interaction,  which  in  accordance  to  Norman’s  model,  would 
occur  after  the  conclusion  of  evaluation  and  before  a  new  goal  is  formed. 


Conclusions  for  IASs: 

1 .  The  authors  advocate  that  Norman’s  model,  which  outlines  seven  stages  of  operator  activity 
(establishment  of  a  goal,  forming  of  an  intention,  specifying  an  action,  executing  this  action  and 
then  evaluation  of  this  action  which  includes  perception,  interpretation,  and  evaluation  of 
results  in  comparison  to  expectation),  identifies  that  adaptation  should  occur  after  the 
conclusion  of  an  evaluation  and  before  a  new  goal  is  formed. 
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Reference: 


Miller,  C.  (2005).  Using  Delegation  as  an  Architecture  for  Adaptive  Automation.  Technical  Report 
(No.  AFRL-H E-WP-TP-2005-0029). 


Overview: 

A  3D  model  framework  for  adaptive  automation,  referred  to  as  "Levels  of  Delegation",  is 
described..  Delegation  implies  that  a  subordinate  is  given  the  responsibility  to  perform  a  task  (with 
its  subtasks),  along  with  some  authority  to  decide  how  to  perform  that  task,  as  well  as  access  to 
resources  with  some  authority  to  decide  how  to  use  them  to  perform  the  task.  This  paper  describes 
the  use  of  this  framework  within  an  application  called  Playbook. 

The  “Delegation  Framework”  has  three  dimensions,  AAA:  Level  of  Authority,  Level  of  Abstraction 
and  Level  of  Aggregation.  These  dimensions  define  a  Delegation  Space  of  human-automation 
relationships  within  which  delegation  occurs  and  can  be  characterized.  The  three  scales  must  be 
used  to  specify  four  variables  which  define  the  delegation  space:  the  level  of  abstraction  and  the 
level  of  authority  on  it,  and  the  level  of  aggregation  and  the  level  of  authority  on  it. 

Below  is  a  description  of  each  dimension: 

•  Levels  of  Authority.  Full,  inform,  override,  approval,  recommend,  monitor,  none. 

•  Level  of  Abstraction.  Automation  can  have  responsibility  for  higher-  or  lower-level  tasks  within 
the  task  hierarchy. 

•  Level  of  Aggregation.  Identifies  how  much  (and/or  which  type)  of  resource  each  actor  is 
authorized  to  use. 


Conclusions  for  IASs: 


1 .  All  three  dimensions  may  not  be  available  or  relevant  to  every  system  or  every  interaction,  but 
the  authors  advocate  that  the  model  needs  to  be  rich  enough  to  encompass  them. 


Reference: 


Moray,  N.,  Inagaki,  T.  &  Itoh,  M.  (2000).  Adaptive  automation,  trust,  and  self-confidence  in  fault 
management  of  time-critical  tasks.  Journal  of  Experimental  Psychology:  Applied,  6,  44-58 


Overview: 

A  study  was  conducted  to  examine  how  supervisory  control  over  automation  influences  fault 
diagnosis  as  well  as  its  effect  on  the  negotiation  that  occurs  between  the  operator  and  the  system. 
Results  are  interpreted  in  relation  to  the  Sheridan-Verplank  scale,  Inagaki's  theory  of  situation 
adaptive  automation,  and  the  work  of  Lee,  Moray  and  Muir  on  human  intervention  in  automated 
systems. 

Sheridan-Verplan  scale:  A  taxonomy  to  describe  who  (human  or  system)  performs  a  particular  task 
and  who  has  control  over  the  allocation  of  that  task. 
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Conclusions  for  IASs: 

1 .  Mode  of  control  (allocation  of  task  and  initiation)  depends  on  the  goal  of  the  operator.  For 
instance,  in  a  fast-paced  environment  that  requires  quick  control,  automation  may  be  more 
appropriate. 

2.  To  decide  which  agent  (human  or  system)  has  final  authority,  the  costs  and  payoffs  with 
different  outcomes  should  be  examined. 

3.  In  at  least  some  time-critical  situations,  overriding  authority  should  be  given  to  system,  and 
appropriate  sensors  and  safe  decision  rules  should  be  implemented  (e.g.,  20-min  rule  in 
nuclear  industry). 

4.  Trust  is  strongly  affected  by  system  reliability,  while  self-confidence  is  not,  at  least  in  systems 
where  operators  can  distinguish  which  tasks  are  accomplished  manually  from  those  performed 
automatically. 

5.  There  is  little  effect  of  unreliability,  if  reliability  is  at  least  90%. 

6.  For  effective  performance  and  fault  management,  adaptive  automation  in  real-time  is 
necessary. 


5. 2.2.6  Designing  the  Nature  of  the  Human-System  Interaction 

Studies  have  examined  the  effects  of  providing  software  agents  with  human-like  attributes 
(e.g.,  etiquette,  personality)  to  improve  the  quality,  and  therefore  the  effectiveness,  of  human- 
system  interactions.  Studies  have  also  investigated  cover  the  social,  political  and  ethical  issues 
relating  to  the  use  of  agents. 


Armentano,  M.,  Godoya,D.  and  Amandi,  A.  (2006).  Personal  assistants:  Direct  manipulation  vs. 
mixed  initiative  interfaces.  International  Journal  of  Human-Computer  Studies  64  (2006)  27-35 


Overview: 

This  paper  explores  new  mixed-initiative  metaphors  to  enhance  an  operator’s  ability  to  directly 
manipulate  interfaces.  Mixed-initiative  interaction  is  referred  to  as  a  flexible  interaction  strategy  in 
which  agents  are  used  to  manage  information  overload.  A  study  evaluating  how  the  interaction 
metaphor  can  affect  the  operator  perception  of  agent  capabilities  is  reported. 

The  mixed-interface  is  the  “PersonalSearcher”,  an  intelligent  agent  that  builds  an  operator  profile 
implicitly  by  observing  operator  behaviour  while  operators  are  performed  regular  activities  on  the 
Web.  An  agent  is  able  to  deduce  the  topics  an  operator  is  interested  in  to  create  an  operator  profile 
by  using  a  content-based  analysis  of  the  information  extracted  by  observation. 

The  study  compared  two  interfaces:  1)  an  operator  interacts  with  the  interface  directly  and  has  no 
control  over  displayed  suggestions  (automation)  and  2)  an  operator  interacts  with  an  animated 
“agent”  instead  of  the  interface  and  has  control  over  suggestions  (mixed-initiative). 

Results  indicate  that  the  mixed-initiative  interface  increased  situational  awareness  (i.e. ,  operators 
noticed  improvements  in  the  agent  suggestions  over  time),  but  that  participants  were  more  critical 
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of  suggestions. 


Conclusions  for  IASs: 

1 .  Mixed-initiative  interfaces  (e.g.,  direct  interaction  with  an  agent)  can  increase  situational 
awareness  and  develop  a  better  mental  model  of  the  system. 

2.  Designers  must  be  careful  when  designing  mixed-initiative  interfaces  to  ensure  a  proper 
mental  model  of  the  system  is  achieved. 


Sheridan,  T.B.,  and  Parasuraman,  R.  (2006).  Human-automation  interaction.  In  R.S. Nickerson 
(Ed.).  Reviews  of  Human  Factors  and  Ergonomics,  Volume  1.  HFES:  Santa  Monica,  CA. 


Overview: 

This  paper  reviews  recent  research  in  the  area  of  human-automation  interaction.  It  describes 
taxonomies  including  supervisory  control  of  automation  and  function  allocation,  and  models  of 
human-automation  interaction.  The  paper  outlines  automation-related  accidents  associated  with 
inadequate  feedback  and  misuse  of  automation,  and  evaluates  the  social,  political,  and  ethical 
issues  related  to  role  of  etiquette  and  trust  on  operator  performance. 

Operator-automation/aqent  interaction:  The  authors  outline  three  ways  that  an  operator  can 
interact  with  a  system: 

4.  By  specifying  to  the  automation/agent  the  task  goals  and  constraints  and  possible  trade-offs 
(e.g.,  pilots  programming  flight  management  systems); 

5.  By  controlling  the  automation/agent  to  start,  stop  or  modify  the  execution  of  the  automatic  task 
(e.g.,  clock  time;  abort  automatic  execution);  and, 

6.  By  receiving  information,  energy,  physical  objects,  or  substances  from  the  automation/agent, 
(e.g.,  warning  or  alarm  display;  expert  system  giving  advice). 

Supervisory  control  over  automated  systems:  A  new  relationship  between  the  operator  and  the 
system  is  identified,  whereby  the  operator  supervises  an  intelligent  but  subordinate  system  by 
issuing  instructions,  and  the  subordinate  executes  those  instructions  by  using  the  system’s  own 
memories,  built-in  programs,  sensors  and  energy  sources. 

Delegation  interfaces:  In  these  systems,  the  operator  delegates  tasks  to  the  system,  at  times  of  the 
operator’s  own  choosing  and  receives,  feedback  on  their  performance. 


Conclusions  for  IASs: 

1 .  The  role  of  social  etiquette.  The  authors  found  that  implicit  codes  of  behaviour  between 
individuals  in  a  social  setting  may  also  play  a  key  role  in  operator-system  relations.  For 
instance,  it  should  not  be  assumed  that  every  operator  is  the  same;  the  machine  should  be 
sensitive  and  adapt  to  the  individual,  cultural,  social,  and  contextual  differences.  In  addition, 
automation  should  be  presented  following  good  ‘etiquette’  (social  mores  that  apply  to  people 
may  also  apply  to  intelligent  systems). 

2.  Proper  feedback  anc[  communication.  Systems  should  provide  the  operator  with  information 
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concerning  automation  modes,  system  states  and  future  automated  actions.  This  may  improve 
human-machine  communication,  and  therefore  potentially  enhance  system  performance. 

3.  Ecological  Interface  Designed  displays  for  non-routine  conditions.  EID  displays  might  be 
particularly  helpful  for  operator-automation  interaction  under  non-routine  conditions.  When 
unexpected  events  occur,  an  operator-automation  display  interface  is  required  that  allows  for 
quick  comprehension  of  system  state  and  rapid  action. 

4.  The  authors  note  that  one  of  the  main  problems  of  operator-system  automation  is  not  one  of 
authority  or  autonomy,  but  of  cooperation  and  observability;  cooperation  means  a  shared 
mental  representation  between  the  operator  and  the  system. 

5.  The  authors  identify  the  “mixed-initiative  problem”;  as  systems  become  more  complex,  more 
control  loops  are  required  and  the  probability  of  these  control  loops  interfering  with  each  other 
increases.  This  can  occur  if  the  operator  cannot  see  what  actions  the  other  control  loops  are 
performing,  or  what  loops  are  being  accessed  from  the  same  resource  pool. 


Gallimore,  J.  J.  and  Prabhala,  S.  (2006).  Creating  Collaborative  Agents  with  Personality  for 
Supervisory  Control  of  Multiple  UCAVs.  In  Proceedings  of  RTO  Human  Factors  and  Medicine 
Panel  (HFM)  Symposium  held  in  Biaritz,  France. 


Overview: 

This  paper  outlines  research  to  develop  a  systematic  approach  for  investigating  the  development 
of  computer  agents  with  personality  as  a  means  of  improving  collaboration  between  humans  and 
automation  in  a  UCAV  control  task. 

The  authors  define  agent  personality  as  a  system  that  interacts  with  the  operator  by  adhering  to 
human  modes  of  communication  (i.e.,  action,  language,  and  behavior).  To  investigate  possible 
interaction  modes,  a  discrete  simulation  was  developed  for  operators  to  interact  with  two  computer 
agents,  with  differing  ‘personalities’  in  a  UCAV  supervisory  control  task.  One  agent  was  modeled  to 
be  high  in  extroversion,  agreeableness,  conscientiousness,  intellect,  and  emotional  stability  (i.e., 
low  neuroticism).  The  other  was  modeled  to  be  lower  on  these  dimensions.  A  third  condition 
included  a  system  with  no  personality.  For  example,  in  CAP-A,  the  computer  agent  greets  the 
human  operator  by  specifically  calling  them  by  their  name  in  a  friendly  tone,  whereas  CAP-B 
greets  the  human  operator  by  just  saying  hello  in  a  monotone  voice.  The  no-personality  condition 
gives  no  verbal  greeting. 


Conclusions  for  IASs: 

1 .  Although  not  significant,  results  indicate  that  humans  do  perceive  personality  in  the 
collaborative  computer  agents  and  human  performance  was  enhanced  when  they  were 
incorporated  into  a  UCAV  supervisory  control  task. 
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Lee,  K.M.  and  Nass,  C.  (2003).  Designing  social  presence  of  social  actors  in  human  computer 
interaction.  Proceedings  of  CHI  2003,  April  5-10  2003,  Fort  Lauderdale,  FL. 


Overview: 

This  paper  outlines  a  study  to  examine  the  effects  of  interaction  between  operator  factors  and 
media  factors  on  feelings  of  “social  presence”.  In  this  paper,  the  authors  explore  the  effects  of 
personality  on  feelings  of  social  presence.  To  demonstrate  the  strength  of  these  effects,  they 
explore  the  use  of  synthetic  speech. 

Results  from  the  study  found  that  operators  had  automatic  social  responses  to  artificial 
representations  with  “humanistic  properties”  such  as  language  and  personality. 

•  Operators  (especially  extrovert  operators)  felt  stronger  social  presence  when  they  heard  a 
computer  voice  manifesting  a  personality  similar  to  their  own  than  when  the  voice  did  not 
match  their  personality,  even  when  the  voices  were  clearly  synthetic  (similarity  effect). 

•  Both  of  the  experiments  showed  that  a  voice  suggesting  an  extrovert  personality  induced  a 
greater  sense  of  social  presence  than  a  voice  that  sounded  like  an  introvert. 

While  still  an  open  question,  current  research  suggests  that  humans  have  an  automatic  tendency 
to  be  very  liberal  in  assigning  humanity  to  an  artificial  stimulus,  as  long  as  they  have  at  least 
minimal  human  features  and  if  social  rules  governing  human-to-human  interaction  Are  followed. 


Conclusions  for  IASs: 

1 .  Results  from  the  study  indicate  that  customization  of  a  computer  voice  according  to  an 
operators’  personality  will  increase  feelings  of  social  presence. 


Sofge,  D.,  Bugajska,  M.,  Adams,  W.,  Perzanowski,  D.,  and  Schultz,  A.  (2003).  Agent-based 
Multimodal  Interface  for  Dynamically  Autonomous  Mobile  Robots,  Technical  Report  prepared  for 
Navy  Center  for  Applied  Research  in  Artificial  Intelligence,  Naval  Research  Laboratory, 
Washington,  DC. 


Overview: 

An  agent-based  multi-modal  interface  is  presented  that  was  designed  as  a  means  for  the 
robot/operator  to  request  information  through  a  “natural  language  interface”  that  uses  combined 
speech  recognition  and  a  gesture  interpretation  process,  among  other  command  input  modes.  The 
dynamic  allocation  of  tasks  is  based  on  a  goal/spatial  relation  architecture. 

The  authors  define  “Human-centric”  as  a  system  that  focuses  on  the  needs  and  natural  modes  of 
interaction  of  the  operator  rather  than  the  robot.  A  key  feature  of  the  interface  is  the  use  of  multiple 
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overlapping  (and  sometimes  redundant)  modes  of  communication  between  the  operator  and  the 
robot.  These  are  overlapping  (and  sometimes  redundant)  modes  of  communication  that  provide 
the  operator  with  a  natural  interface  to  the  system,  allowing  the  operator  to  choose  the  mode  of 
communication  most  comfortable  to  him/her  given  the  current  task,  situation  and  environmental 
conditions.  To  control  the  robots  through  the  autonomous  robot  agent,  the  operator  interfaces  with 
the  Robot  Interface  Client. 

Agents  provide  a  natural  and  flexible  means  for  integrating  multiple  interface  modules  together. 


Conclusions  for  IASs: 

1.  Agent-based  architecture  can  provide  a  natural  and  scalable  approach  to  implementing  a 
multimodal  interface  to  control  mobile  robots  through  dynamic  automation. 

2.  Direct  communication  with  an  agent  through  an  interface  (i.e.,  natural  language  and 
gestures)  can  be  an  effective  means  of  human-machine  communication^ 


Serenko,  A.  (2007).  Are  interface  agents  scapegoats?  Attributions  of  responsibility  in  human-agent 
interaction.  Interacting  with  Computers,  19,  293-303 


Overview: 

This  paper  presents  research  to  understand  the  behavior  of  interface  agent  operators.  Several 
conclusions  are  presented  for  understanding  operator-system  interaction. 

Interface  agents:  The  authors  describe  interface  agents  “as  an  intermediary  between  a  person  and 
various  components  of  a  computer  system”.  They  are  used  as  a  communication  tool.  Researchers 
have  begun  experimenting  with  the  incorporation  of  animated,  human,  or  cartoon-like  interface 
agents  in  GUIs. 

Social  psychology  framework  for  human-agent  interaction :  The  authors  recommend  the  use  of 
social  rules,  behaviours  and  expectations  for  interface  agents  to  mimic  social  principles  of  human- 
human  communication. 


Conclusions  for  IASs: 

1 .  Interface  agents  can  be  used  to  emphasize  the  autonomy  of  software  systems. 

2.  The  area  of  social  psychology  offers  a  strong  theoretical  framework  that  may  be  successfully 
utilized  to  study  human-agent  interaction. 

3.  Agent  designers  should  be  aware  that  the  more  autonomous  interface  agents  become,  the 
more  responsibility  operators  will  assign  to  agents  if  they  fail  to  deliver  what  is  expected. 

4.  When  an  agent  that  possesses  a  high  degree  of  autonomy  helps  a  person  complete  a 
computer-related  task  successfully,  an  individual  is  willing  to  acknowledge  the  contribution  of 
the  agent. 

5.  Life-like  agents  may  trigger  natural  behavior  in  operators. 
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Prendinger,  H.,  Ma,  C.,  Ishizuka,  M.  (2007).  Eye  movements  as  indices  for  the  utility  of  life-like 
interface  agents:  A  pilot  study.  Interacting  with  Computers,  19,  281-292. 


Overview: 

This  paper  outlines  a  pilot  study  that  compared  three  types  of  media:  an  animated  agent,  a  text 
box,  and  speech  only.  Authors  propose  a  different  approach  to  evaluating  animated  agents,  one 
that  is  based  on  eye  movement  behavior  of  operators  interacting  with  the  OMI.  The  operators  do 
not  manipulate  the  interface.  The  authors  argue  that  operators  merely  watching  a  presentation 
interact,  even  involuntarily,  with  their  eye  movements. 

Physiological  signals  as  an  evaluation  method  for  OMIs  and  as  an  input  modality:  The  eye 
movement  data  was  analyzed  for  diagnostic  use  (as  a  means  to  examine  the  operator’s  attention 
to  evaluate  the  usability  of  interfaces),  and  for  interactive  use  (a  system  responds  to  the  observed 
eye  movements  and  can  thus  be  seen  as  an  input  modality). 

The  investigation  of  eye  movements  revealed  that  deictic  gestures  by  the  agent  are  more  effective 
in  directing  the  attentional  focus  of  subjects  to  relevant  interface  objects  than  the  media  used  in  the 
two  control  conditions,  at  a  slight  cost  of  distracting  the  operator  from  visual  inspection  of  the 
object  of  reference.  The  results  also  demonstrate  that  the  presence  of  an  interface  agent 
seemingly  triggers  natural  and  social  interaction  protocols  of  human  operators. 


Conclusions  for  IASs: 

1 .  Physiological  signals  can  be  used  as  an  evaluation  method  for  OMIs  and  as  an  input  modality. 

2.  Eye  movement  data  might  offer  valuable  information  relevant  to  the  utility  of  life-like  agents. 

3.  The  usability  of  interfaces  can  be  assessed  using  eye  movements. 

4.  The  tracking  of  eye  movements  can  capture  an  operator’s  interaction  with  the  system  in  real 
time,  which  is  hard  to  accomplish  using  post-experiment  questionnaires. 


5.2.3  Example  of  Human-System  Teamwork  -  Tasking  Interface 
Manager  (Cognitive  Cockpit) 

The  Cognitive  Cockpit  project  included  research  and  development  of  a  cockpit  interface  sub¬ 
system  for  managing  the  outputs  of  the  pilot  aiding  system,  the  pilot  cognition  monitoring 
system,  and  the  interaction  with  the  task  automation.  The  most  effective  manner  of  ensuring 
that  the  pilot  retains  control  of  the  mission  at  all  times  is  to  allow  the  pilot  to  interact  with  the 
automation  and  aiding  through  a  Tasking  Interface  Manager.  A  TIM  is  considered  to  be 
necessary  for  the  fully  integrated  system  to  ensure  that  the  pilot’s  tasks,  workload  and  cockpit 
control/display  interfaces  are  managed  effectively  (see  Bonner,  et  al.,  2000). 

The  TIM  is  based  on  the  Army’s  Rotary  Pilot’s  Associate  programme  and  the  Air  Force’s 
Pilot’s  Associate  programme  (Miller,  Pelican,  and  Goldman,  1999).  A  tasking  interface 
allows  a  pilot  to  task  automation  in  the  same  manner  that  an  intelligent,  knowledgeable, 
subordinate  crewmember  might  be  tasked.  It  will  require  the  development  of  an  intuitive 
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cockpit  control/display  interface  that  provides  the  required  pilot  control  of  levels  of  task 
automation. 


The  TIM  utilises  information  from  the  monitoring  and  analysis  of  the  mission  tasks  (i.e., 
SASS)  and  from  pilot  state  monitoring  (i.e.,  COGMON)  to  drive  intelligent  adaptive 
automation  and  information  presentation  (i.e.,  COGPIT),  as  well  as  task  and  timeline 
management  in  accordance  with  the  requirements  of  the  mission  plan.  The  functional 
architecture  of  the  TIM  comprises  twelve  functions,  with  a  flow  of  information  and  control 
across  the  functions  as  illustrated  by  Figure  9.  The  arrows  represent  the  flow  of  information 
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across  functions;  the  solid  arrows  represent  primary  information  and  the  dashed  arrows 
represent  secondary  information. 

Figure  9:  Functional  Architecture  of  the  Tasking  Interface  Manager  (from  Bonner, 

Taylor,  Fletcher  and  Miller,  2000). 


The  functional  architecture  of  the  TIM  affords  a  system’s  four  main  capabilities: 

1.  Shared  Task  Model  In  order  for  the  TIM  to  be  able  to  determine  information  and 
automation  needs,  the  state  of  the  mission  plan  needs  to  be  known.  This  involves  tracking 
the  tasks  that  are  occurring.  In  order  to  achieve  this,  it  is  essential  that  an  operator’s  goals 
and  plans  be  encoded  and  tracked,  and  that  the  model  of  current  and  planned  tasks  is 
dynamically  modified  to  keep  pace  with  unfolding  events.  The  use  of  a  task  model,  shared 
by  both  the  operator  and  the  knowledge-based  planning  system,  affords  a  high  level  of  co¬ 
ordination  between  the  operator  and  the  system; 

2.  Task  Tracking.  A  key  capability  of  the  TIM  identifies  the  need  for  a  full  goal/plan 
tracking  capability,  which  allows  the  system  to  track  any  task  undertaken  by  the  operator; 
specifically  those  tasks  that  are  instantiated  in  the  mission  plan.  There  are  two  critical 
requirements  of  any  goal/plan  tracking  system;  it  must  be  explicit  (i.e.,  visible  to  the 
operator)  and  interactive  (i.e.,  the  operator  must  be  able  to  directly  input  or  over-ride 
tasks). 

3.  Communication  of  Intent.  Another  capability  of  the  TIM  is  to  allow  the  operator  to 
interact  with  advanced  automation  flexibly  at  a  variety  of  levels.  This  allows  the  operator 
to  smoothly  vary  the  amount  of  automation  used  depending  on  variables  such  as  time 
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available,  workload,  criticality  of  the  decision,  and  degree  of  trust;  these  variables  are 
known  to  influence  human  willingness  and  accuracy  in  automation  use.  There  are  three 
primary  challenges  involved  in  the  construction  of  a  TIM: 

a.  A  shared  vocabulary  must  be  developed,  through  which  the  operator  can  flexibly 
delegate  tasks  to  the  automation,  and  the  automation  can  report  how  it  intends  to 
perform  those  tasks; 

b.  Sufficient  knowledge  must  be  built  into  the  interface  to  enable  making  intelligent 
choices  within  the  tasking  constraints  imposed  by  the  operator.  This  is  the  role  of 
the  information,  and  automation  needs  interpreters  as  illustrated  in  Figure  1;  and, 

c.  One  or  more  interfaces  must  be  developed  which  will  permit  inspection  and 
manipulation  of  the  tasking  vocabulary  to  delegate  tasks  and  review  task 
elaborations  in  a  rapid  and  easy  fashion.  The  goal  is  to  allow  the  operator  to 
communicate  tasking  instructions  in  the  form  of  desired  goals,  tasks,  partial  plans 
or  constraints  in  accordance  with  the  task  structures  defined  in  the  shared  task 
model.  For  example,  Miller  (2005)  developed  prototype  tasking  interfaces  based 
on  a  playbook  metaphor  (Section  8.3.3)  whereby  a  set  of  available  plans  are 
described  and  visualised  in  a  comparatively  limited  vocabulary  of  previously 
defined  ‘plays’  that  can  then  be  adapted  rapidly  to  the  current  context.  The  TIM 
uses  a  variation  of  the  playbook  metaphor. 

4.  Adaptive  Aiding  and  Automation.  Analysis  of  the  requirement  for  an  operator  to  authorise 
and  control  automation  levels  through  the  TIM  has  led  to  the  development  of  the  Pilot 
Authorisation  and  Control  of  Tasks  system.  The  PACT  system  uses  military  terminology 
(i.e.,  Under  Command,  At  Call,  Advisory,  In  Support,  Direct  Support,  and  Automatic)  to 
distinguish  realistic  operational  relationships  for  five  aiding  levels,  with  progressive  pilot 
authority  and  computer  autonomy,  supporting  situation  assessment,  decision  making  and 
action  (Figure  10).  The  operator  is  able  control  this  allocation  through  the  following: 

a.  Pre-set  operator-preferred  defaults; 

b.  Operator  selection  during  pre-flight  planning; 

c.  Changed  by  the  operator  during  in-flight  re -planning;  and, 

d.  Automatically  changed  according  to  operator  agreed,  context-sensitive  adaptive 
rules. 
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AUTOMATION  PILOT 


PACl 

AUTHORITY 

AUTOMATIC 

B 

Interrupt 

DIRECT  SUPPORT 

D 

Revoking  action 

IN  SUPPORT 
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Acceptance  of  advice 
&  authorising  action 

ADVISORY 

B 

Acceptance 
of  advice 

AT  CALL 

B 

Advice,  only 
if  requested 

COMMAND 

B 

Pilot  full 
authority 

COMPUTER 

AUTONOMY 


Full  computer 
autonomy 

Action, 

unless  revoked 
Advice, 

and  if  authorised, 

Advice 


Advice,  only 
if  requested 

Pilot  full 
authority 


Figure  10:  Summary  of  PACT  levels  (from  Bonner  et  al.,  2000). 


5.2.4  Design  Issues  for  ‘Real-time’  Systems 

The  use  of  an  IAS  in  many  military  contexts  (e.g.,  aviation)  requires  aiding  and  interaction  in 
real  time.  In  real-time  operations  the  correctness  of  a  system  is  dependent  not  only  on  the 
correctness  of  its  result,  but  also  on  meeting  stringent  timing  requirements.  The  deadlines  for 
tasks  that  a  real-time  system  must  perform  can  be  characterised  as  hard,  firm,  or  soft.  If  a  hard 
deadline  is  not  met  then  the  consequences  are  usually  disastrous.  Failure  to  meet  a  firm 
deadline  means  that  the  results  of  the  computation  have  no  utility.  Soft  deadlines  mean  the 
results  of  the  computation  are  still  useful  after  the  deadline  has  elapsed,  but  have  decreasing 
utility  as  a  function  of  the  time  elapsed  (Hayes-Roth,  1991). 

The  following  requirements  for  the  real-time  operation  of  IASs  can  be  used  in  addition  to 
those  in  the  previous  section  (Hayes-Roth,  1991). 

5. 2.4.1  Cognitive  Versatility 

•  Multi-faceted  expertise.  The  IAS  should  be  able  to  perform  different  types  of 
reasoning  tasks  in  an  attempt  to  solve  problems  in  a  variety  of  domains  utilising  a 
number  of  problem-solving  techniques; 

•  Concurrent  reasoning  activities.  The  IAS  must  be  capable  of  simultaneous  reasoning 
about  a  number  of  concurrent  activities; 

•  Incremental  reasoning.  The  IAS  must  be  able  to  integrate  information  over  time  to 
produce  an  accurate  assessment  of  the  current  situation;  and, 

•  Explanation.  The  IAS  should  be  able  to  explain  all  aspects  of  its  behaviour  in  the  time 
available. 
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5. 2.4. 2  Management  of  Integration 

•  Functional  asynchrony  and  parallelism.  The  IAS  must  investigate  anomalies,  and 
also  perform  routine  actions  within  specified  time  limits; 

•  Continuous  operation.  The  IAS  must  be  capable  of  functioning  over  extended  time 
periods;  and, 

•  Functional  Integration.  The  IAS  should  be  able  to  perform  accurate  reasoning  even 
where  certain  conditions  affect  normal  output  of  the  reasoning  process  (e.g., 
recommendations  may  differ  as  a  function  of  weapons  fit). 

5. 2.4. 3  Management  of  Complexity 

•  Selective  attention.  The  IAS  may  encounter  situations  in  which  it  cannot  process  all 
the  data  in  real-time.  Therefore,  the  IAS  must  be  able  to  make  choices  about  which 
data  are  the  most  important  and  disregard  extraneous  data.  However,  it  is  imperative 
that  the  IAS  still  be  alert  to  new  data  that  might  be  critical  to  the  current  task; 

•  Automatic  performance.  The  IAS  must  be  able  to  deal  with  complex  anomalies  or 
situations  whilst  performing  important  routine  activities  in  a  timely  manner;  and, 

•  Focused  reasoning.  The  IAS  must  be  able  to  control  its  reasoning  such  that  it  can 
achieve  specific  goals.  The  IAS  will  face  more  ‘problems’  than  it  can  solve  in  real 
time,  and  so  it  is  important  that  the  IAS  must  be  able  to  choose  the  most  urgent 
problem(s)  to  solve  first. 

5. 2.4.4  Real-time  Performance 

•  Guaranteed  inter-operation  latencies.  The  IAS  must  be  able  to  guarantee  that  it  can 
achieve  certain  goals  within  the  prescribed  time  frames; 

•  Time-stress  responsivity.  The  IAS  should  be  able  to  respond  to  increased  pressure  on 
time  resources  by  decreasing  its  response  latency; 

•  Graceful  degradation.  The  IAS  must  be  able  to  reduce  response  latency  as  a  function 
of  time  stress  by  compromising  precision  and  confidence  in  a  graduated  manner;  and, 

•  Speed-knowledge  independence.  The  IAS  must  be  able  to  produce  consistent  response 
latencies  despite  the  inclusion  of  new  knowledge. 

From  these  guidelines,  it  is  evident  that  there  are  a  number  of  obstacles  in  a  real-time  IAS 
providing  outputs  with  guaranteed  levels  of  completeness,  timeliness,  precision  and 
confidence  for  a  wide  range  of  input  states.  However,  a  number  of  issues  arise  if  these 
obstacles  cannot  be  overcome  by  increasing  the  computing  power  available,  or  bounding  the 
scope  of  the  system’s  operations  (Hayes-Roth,  1991),  including: 

•  What  is  the  effect  on  the  ability  of  the  operator  to  make  decisions  based  upon 
information  of  varying  quality?; 

•  An  operator  may  need  to  identify  why  the  required  level  of  accuracy  could  not  be 
achieved  as  this  information  may  affect  future  decisions;  and 
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•  The  interface  should  provide  as  much  information  as  possible  about  any  problem  that 
could  not  be  solved,  so  that  the  operator  can  judge  the  best  course  of  action  in  the 
situation. 


5.3  Summary 

Table  15  summarises  issues  relating  to  adaptive  automation  and  interfaces,  Table  16 
summarises  operator-system  teamwork  implications  and  supervisory  control  for  Intelligent 
Adaptive  Systems,  and  Table  17  provides  a  summary  of  goal/task  based  adaptive  automation 
and  dynamic  learning  systems. 
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Table  15:  Summary  of  issues  relating  to  Adaptive  Automation  and  Adaptive  Interfaces. 


Operator  Training 

Learning  Systems  (Dynamic 
Adaptation) 

Information  Presentation 

What? 

The  operator  should  be  on  the  features  and 
functionality  of  the  system. 

The  system  learns  (modifies  its  algorithms)  from 
interactions  with  the  operator. 

The  way  information  is  presented  to  the  operator  and 
the  layout  of  the  OMI. 

Debate 

How  much  training  is  required. 

Novice  vs  expert  training. 

What  are  the  criteria  for  system  learning  (i.e., 
operator  behaviour?). 

Dynamic  vs  fixed  implementation  of  adaptation. 

How  to  structure  the  information  and  layout  of  the 

OMI. 

Best  means  to  acquire  requirements. 

Why? 

To  ensure  proper  mental  model  of  the 
system  is  acquired. 

Manage  OOTL  performance  problems  and 
workload. 

Operator  acceptance. 

Particularly  important  in  life-critical, 
mission-critical  systems. 

There  is  a  critical  need  for  operators  to 
understand  and  experience  the  potential 
benefits  of  adaptation. 

A  hybrid  adaptive  OMI  based  on 
experience  with  the  adaptive  system  is 
suggested  to  increase  the  operator’s 
understanding  of  the  system  and  its 
impact:  a  phase  dependent  mix  between 
fully  automatic  and  operator-controlled 
adaptation. 

Powerful  mean  to  adapt  automation  or  Ul  to  the 
individual. 

Support  operator  learning  (of  the  system  and  the 
domain)  and  decision  making. 

Support  dynamic  nature  of  complex 
environments  (e.g.,  fast  changing 

WWW;  complex  and  dynamic  military  operators). 

Ensure  that  the  system  is  “aware”  of  the  world. 

The  OMI  should  present  information  that  allows 
optimal  performance  when  monitoring  automation 
(e.g.,  multiple  robots).  The  layout  of  the  OMI  should 
be  based  on  operator  requirements. 

EID  can  be  used  as  a  hierarchy  tool  to  determine  the 
structure  of  the  OMI. 

An  intelligent  decision  aid  should  support  the  level  of 
experience  and  skill  of  the  decision  maker  (e.g., 
present  information  according  to  experience  level). 

Skill  reduction  in  the  operator. 

Failure  of  the  operator  to  attend  to  important 
situational  cues. 

Ul  should  be  predictable  to  support  operator  trust  and 
promote  rapid  learning. 

Reference 

Kaber,  Wright,  Prinzel,  &  Clamann,  (2005); 
Schneiderman  and  Maes  (1997);  Hou,  M. 
Kobierski,  R.,  Herdman,  C.  (2006) 

Miller,  C.A.,  Dorneich,  M.C.  (2006);  Oppermann, 
Rashev,  &  Kinshuk.  (1997);  Roberts  (2006); 
Schneiderman  and  Maes  (1997);  Hou,  M. 

Kobierski,  R.,  Herdman,  C.  (2006) 

Kirlik,  Markert,  &  Kossack.  (1992);  Hou,  M.  Kobierski, 

R.,  Herdman,  C.  (2006) 
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Authority  (i.e.,  Control) 

Allocation  of  Task/Function 

OOTL  performance  problems 

What? 

Who,  the  operator  or  the  system,  should 
have  authority  of  initiation  of  the  adaptation. 

There  are  currently  two  main  philosophies: 
the  operator  should  “supervise  and  delegate” 
adaptation  or  that  the  authority  could  be 
shared  among  the  operator  and  the  system. 

The  amount  of  operator  control  and 
involvement  (adaptivity  and  automation)  and 
the  roles  that  humans  and  machines  engage 
in  or  are  in  control  over  (i.e.,  information 
processing)  should  be  considered  in  the 
design  of  a  system. 

The  allocation  of  automated  tasks  (to  the  operator 
or  the  agent)  at  particular  stages/roles  based  on 
task/function  and  context. 

The  inability  to  intervene  or  assume  the  task  or  role  that 
the  automatic  system  was  responsible  for  at  the  time. 
When  the  operator  is  kept  out-of-the-loop  they  are 
slower  at  detecting  errors,  less  likely  to  be  able  to 
intervene,  lose  skills  that  were  previously  completed 
manually,  and  are  unable  to  fully  understand  the 
systems’  status. 

Debate 

How  much  each  agent  should  have  control 
and  at  what  stage? 

Who  makes  that  decision? 

Should  operators  always  be  in  control? 

What  is  the  best  strategy  to  determine  which  agent 
should  perform  the  task  /  function? 

Balance  between  workload  and  OOTL  performance 
problems. 

How  much  of  this  balance  relies  on  operator  control  of 
the  system. 

Why? 

Keep  operator  in  the  loop. 

Operator  acceptance. 

Trust. 

Cultural  aspects. 

Safety. 

There  is  a  spectrum  of  operator  control  over 
the  adaptation  of  a  system. 

Need  a  balance  between  workload  and 
human  OOTL  performance  problems. 

Operator  control  can  lead  to  better  perceived 
performance  and  higher  overall  satisfaction. 

Knowledge  of  the  task  context  and  its  associated 
information  needs  can  help  in  the  development  of 
overall  system. 

Implementation  of  adaptation. 

Managing  task  demand. 

Increasing  operator  performance. 

Automated  information  acquisition,  analysis  and 
action  implementation  can  help  reduce  workload 
and  increase  performance  in  ATC  tasks. 

Adaptive  allocation  could  produce  positive  benefits 
to  a  wide  range  of  pilot  functions  including  task 
prioritization,  mission  segmenting,  task  initiation 
and  cessation,  risk  identification,  and  workload 
management. 

It  is  critical  that  the  system  informs  the  operator  of  any 
changes  on  the  interface. 

Lack  of  proper  training. 

Lack  of  operator  control. 

Reference 

Miller  &  Dorneich  (2006);  Albery  & 

Khomenko  (2002);  Oppermann,  Rashev,  & 
Kinshuk.  (1997);  Roberts  (2006);  Kaber, 

Miller  &  Dorneich  (2006);  Kaber,  Wright,  Prinzel,  & 
Clamann,  (2005);  Scallen,  &  Hancock,  (2001). 

Roberts  (2006);  Kaber,  Wright,  Prinzel,  &  Clamann, 

(2005);  Hou,  M.  Kobierski,  R.,  Herdman,  C.  (2006); 
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Authority  (i.e.,  Control) 

Allocation  of  Task/Function 

OOTL  performance  problems 

Wright,  Prinzel,  &  Clamann,  (2005); 

Findlater,  L.,  &  McGrenere,  J.  (2004);  Kirlik, 
Markert,  &  Kossack.  (1992). 

Findlater,  L.,  &  McGrenere,  J.  (2004). 

Communication  and  Interaction 

What? 

How  agents  (operators  and  systems)  communication  and  interact. 

Debate 

Should  agents  (operators  and  systems;  systems  and  systems)  communication  as  humans  do? 

Why? 

Direct  interaction  with  an  agent  (e.g.,  paperplip)  can  increase  situational  awareness  and  develop  better  mental  model  of  the  system. 

Adherence  to  an  accepted  but  frequently  implicit  code  of  behaviour  between  individuals  in  any  social  setting  may  also  play  a  key  role  in  human-computer 
relations.  For  instance,  don’t  assume  every  operator  is  the  same;  the  machine  should  be  sensitive  and  adapt  to  individual,  cultural,  social,  and  contextual 
differences. 

Good  ‘etiquette’  are  social  mores  that  apply  to  people  may  also  apply  to  intelligent  machines.  This  can  improve  human-machine  communication  and  therefore 
potentially  enhance  system  performance. 

Personality  in  the  collaborative  computer  agents  may  enhance  human  performance  within  a  supervisory  control  task. 

Operators  (especially  extrovert  operators)  may  feel  stronger  social  presence  when  they  hear  a  computer  voice  manifesting  a  personality  similar  to  their  own 
than  when  the  voice  did  not  match  their  personality,  even  when  the  voices  were  clearly  synthetic  (similarity  effect). 

Both  of  the  experiments  showed  that  a  voice  suggesting  an  extrovert  personality  induced  a  greater  sense  of  social  presence  than  a  voice  that  sounded  like  an 
introvert. 

While  still  an  open  question,  current  research  suggests  that  humans  have  an  automatic  tendency  to  be  very  liberal  in  assigning  humanity  to  an  artificial  stimulus 
as  long  as  they  have  at  least  minimal  human  features  and  if  follow  a  social  rule  governing  human-to-human  interaction. 

Direct  communication  with  an  agent  through  an  interface  can  be  an  effective  means  of  human-machine  communication. 

One  must  use  anthropomorphic  representation  (e.g.,  Microsoft’s  paper  clip)  with  caution:  it  may  mislead  the  designers,  and  deceive  operators;  it  may  interfere 
with  predictability,  reduce  operator  control,  and  undermines  operators’  responsibility.  An  “invisible”  or  transparent  agent  may  be  more  effective. 

Direct  manipulation  designs  promote  rapid  learning.  It  supports  rapid  performance  and  low  error  rates  while  supporting  exploratory  usage  in  positive  ways. 

Eye  movement  data  might  offer  valuable  information  relevant  to  the  utility  of  life-like  agents. 

The  tracking  of  eye  movements  can  capture  the  operator’s  interaction  with  the  system  in  real  time,  which  is  hard  to  do  using  post-experiment  questionnaires. 

Reference 

Armentano,  Godoya,&  Amandi.  (2006);  Sheridan,  &  Parasuraman.  (2006);  Lee  &  Nass.  (2003);  Gallimore  &  Prabhala.  (2006);  Sofge,  D.,  Bugajska,  M.,  Adams, 
W.,  Perzanowski  &  Schultz.  (2003);  Schneiderman  and  Maes  (1997);  Prendinger,  Ma  &  Ishizuka  (2007) 
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Table  16:  Summary  of  operator-system  teamwork  implications  and  supervisory  control  for  Intelligent  Adaptive  Systems. 


OPERATOR-SYSTEM  TEAMWORK  IMPLICATIONS 

Authority  (i.e.,  Control) 

Allocation  of  Task/Function 

Applicability 

Issues 

In  team  coordination,  effective  authority 
coordination  requires  that  cooperating  team 
to  team  members  should  know  as  much  as 
possible  from  each  other’s  performance 
characteristics  and  behavioral  traits. 

The  actual  adjustment  of  an  agent’s  level  of 
autonomy  could  be  initiated  either  by  a 
human,  an  agent,  or  some  third  party. 

A  system  in  which  the  machine  learner  and 
human  operator  more  equally  share 
responsibility  can  guide  the  learning 
process. 

Bayesian  Belief  Networks  can  be  used  to  structure 
task  allocation,  task  sharing,  team  composition, 
procedures  and  OMIs  from  a  teamwork  perspective 

Must  be  careful  of  “clumsy  automation”:  the 
erroneous  notion  that  automation  activities  simply 
can  be  substituted  for  human  activities  without 
otherwise  affecting  the  operation  of  the  system. 

Humans  also  need  to  be  able  to  control  and  re¬ 
direct  the  software  agents  as  task  requirements 
change. 

Teamwork  concepts  should  be  considered  when 
designing  systems  that  require  human  operators  to 
simultaneously  control  several  functions  at  various 
working  positions,  such  as  controlling  multiple  UAVs. 

To  ensure  that  interactions  between  agents  and 
people  are  as  natural  and  effective  as  possible 

The  more  the  system  understands  its  operators  and 
their  tasks,  the  more  useful  it  will  be  for  them. 

Human-machine  cooperation  can  be  achieved  by 
allowing  an  operator,  in  executing  her  part  of  a  plan, 
to  expect  a  system  to  help  in  executing  part  of  the 
plan.  A  plan  is  a  set  of  processes  (often  to  be 
executed  by  a  number  of  different  agents)  that  when 
run  together  successfully,  accomplish  some  goal. 

Plans  attempt  to  express  a  common  understanding 
of  how  an  operator  and  a  system  interact. 

A  mixed-initiative  framework  in  which  the  learner  and 
human  operator  are  each  participants  in  a  dialogue 
could  improve  the  learner's  hypothesis  with  minimal 
effort  on  the  part  of  the  operator. 

A  systems  needs  to  support  multiple  facets  of 
individual  cognitive  and  collaborative  work. 

Reference 

Onken  (2002);  Wolfman,  Lau  Pedro 
Domingos,  &  Weld.  (2001); 

de  Reus,  Roessingh,  &  Pouw  (2006);  Eggleston, 
Roth,  Scott.  (2003); 

de  Reus,  Roessingh,  &  Pouw  (2006);  Franklin, 

Budzik,  &  Hammond.  (2002).;  Wolfman,  Lau  Pedro 
Domingos,  &  Weld.  (2001);  Eggleston,  Roth,  Scott. 
(2003); 
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Situation  Awareness 

Issues 

Each  actor,  both  human  and  agent,  must  be  able  to  realistically  assess  the  overall  situation  and  current  the  state  and  intentions  of  the  other  team 
members. 

The  system  should  always  tries  to  understand  the  operator’s  actions  in  the  context  of  what  it  believes  the  operator  is  doing. 

Automated  agents  need  to  be  observable  (or  transparent/visible)  so  that  operators  are  able  to  see  what  the  automated  agents  are  doing  and  understand 
what  they  will  do  next  relative  to  the  state  of  the  task.  How  much  “visibility”  needed  is  questionable  (i.e.,  not  at  all  but  then  issue  of  trust  and  mistrust  can 
occur  or  fully  visible  such  as  the  Microsoft  “Paperclip”  which  takes  advantage  of  assistant  and  subordinate  metaphors) 

Reference 

Franklin,  Budzik,  &  Hammond.  (2002);  Eggleston,  Roth,  Scott.  (2003); 

SUPERVISORY  CONTROL  IMPLICATIONS 

Authority  (i.e.,  Control) 

Allocation  of  Task/Function 

Applicability 

Issues 

The  tasking  interface  allows  the  operator 
to  inspect  and  interact  with  the  task 
model  by  extending  the  operator’s  ability 
to  “call  plays”  and  activating  tasks  at 
various  levels  of  decomposition  (e.g., 
Playbook). 

Teaching  the  operator  to  become  an 
effective  supervisor. 

Levels  of  Authority:  full,  inform,  override, 
approval,  recommend,  monitor,  none. 

The  Level  of  Aggregation  identifies  how 
much  (and/or  which  type)  of  resource 
each  actor  is  authorized  to  use. 

Tasked  systems  are  always  sub-ordinate,  but 
know  enough  about  the  tasks  in  the  domain  that 
instructing  them  is  vastly  easier  than  instructing 
traditional  automated  systems. 

Level  of  Abstraction.  Automation  can  have 
responsibility  for  higher-  or  lower-level  tasks 
within  the  task  hierarchy. 

Enabling  a  system  to  behave  more  like  an 
intelligent  subordinate,  operators  may  be  more 
tolerant  of  their  weaknesses  and  acceptable  of 
their  capabilities  in  a  controlled  setting  (operator 
acceptance). 

Requires  more  direct  interaction  with  the  tasking 
interface. 

Slightly  higher  workload  when  participants  had 
flexible  access  to  both  manual  control  and 
automated  plays. 

Train  the  human  to  adequately  supervise  the 
system  functioning. 

A  question  is  how  to  control  the  delegated  system. 
Not  all  of  three  (level  of  authority,  abstraction  and 
aggregation)  dimensions  may  be  available  or 
relevant  to  every  system  or  every  interaction,  but 
the  model  needs  to  be  rich  enough  to  encompass 
them. 
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Reference 

Miller  &  Goldman  (1999);  Breton  &  Bosse, 

E.  (2002);  Miller,  C.  (2005). 

Miller  &  Goldman,  R.  (1999);  Miller,  C.  (2005). 

Miller  &  Goldman  (1999);  Parasuraman  &  Miller 
(2006);  Breton  &  Bosse,  E.  (2002);  Miller,  C. 

(2005). 

Situation  Awareness 

Issues 

A  supervisor  role  can  be  an  effective  balancing  technique  between  reducing  mental  workload,  attentional  demands,  the  effect  of  fatigue  and  stress  factors 
and  the  probability  of  errors  and  maintaining  situational  awareness. 

These  interfaces  allow  operators  to  respond  effectively  to  unpredictable  changes  and  which  was  associated  with  the  flexibility  afforded  by  delegation 
interfaces. 

Reference 

Parasuraman  &  Miller  (2006);  Breton  &  Bosse,  E.  (2002); 
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Table  17:  Summary  of  Goal/Task-based  Adaptive  Automation  and  Dynamic  Learning  Systems 


Goal/Task-based  Adaptive  Automation 

Dynamic  Learning  Systems 

Advantages 

Modelling  techniques  can  be  implemented  offline  (and  no  real-time 
criticality  issues)  and  easily  incorporated  into  a  rule-based  expert 
system. 

Pre-defined  rules  and  critical  events  triggers  can  be  combined  with 
other  implementation  methods  (e.g.,  the  level  or  type  of  automation  is 
changed  based  on  an  assessment  of  operator  state)  for  a  best  general 
approach. 

The  operator  can  specify  pre-defined  goals  for  the  automated  planning 
system. 

Goal/Task-Based  systems  can  allow  agents  to  dynamically  and  flexibly 
assume  a  range  of  roles  depending  on  the  task  to  be  performed  and 
the  current  situation. 

Can  provide  a  variety  of  rich  interaction  modes  that  enhance  the 
learning  process  on  the  part  of  the  operator  and  the  system. 

The  operator  can  control  the  next  stage  of  discourse,  updating  the 
state  of  the  learner  which  can  then  be  rescored  based  on  the  new  state 
of  the  learner  (operator). 

A  system  based  on  experience  with  the  adaptive  system  can  increase 
the  operator’s  understanding  of  the  system  and  its  impact. 

As  the  system  learns  from  its  interaction  with  the  operator’s  past 
behaviour,  the  system  can  provide  more  accurate  and  timely 
adaptation  (see  StockTrader).  The  ability  to  reason  about  the  activity  of 
an  operator  is  crucial  to  the  implementation  of  an  intelligent  OMI. 

The  pre-defined  user  model  may  not  apply  to  all  operators  (e.g., 
different  usage  patterns),  and  therefore  a  learning  system  can 
dynamically  adapt  based  on  a  usage  pattern  or  interactivity  with  the 
system. 

The  more  the  system  understands  its  operators  and  their  tasks,  the 
more  useful  it  will  be  for  them. 

Compensate  for  individual  characteristics. 

Disadvantages 

Valid  model  is  required  and  different  models  within  the  same  system 
might  give  contrary  decisions  at  particular  moments. 

Potential  for  complex  implementation:  multiple  processes  (agents)  and 
operational  knowledge  representations  (frameworks)  are  necessary  for 
complex  IAS  interactions. 

The  initiation  of  automation  (i.e.,  allocation  of  task  to  a  system)  must  be 
sensitive  to  the  operator’s  combined  tasking  environment,  which 
depends  on  interactions  among  tasks,  the  environment  and  the 
operator  state  (e.g.,  workload). 

Potential  for  poor  user  models. 

Potential  for  complex  implementation  of  agents. 

Relationship  to 
Frameworks 

COGPIT;  RA/RPA;  Playbook;  CAMA;  Co-Pilot  Electronique;  SAWA; 
Work-Centered  Decision  Support;  CASSY;  DRDC  UAV  Project. 

Intelligent  Classroom;  DIAManD;  StockTrader;  Lookout;  Personal  Web 
Server. 
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6  Operator-State  Monitoring  Approaches 


6.1  Introduction 

An  operator  is  a  critical  determinant  that  influences  the  behaviour  of  a  war-fighting  system. 
Any  minor  errors,  lapses  of  attention  or  losses  of  Situation  Awareness  can  quickly  onset 
disastrous  consequences.  For  this  reason,  the  ability  to  make  inferences  about  current  operator 
state  is  a  critical  requirement  for  effective  system  management.  Such  inferences  would 
provide  information  about  current  operator-state,  but  could  also  signal  possible  problems  at 
later  times  or  for  specific  future  tasks.  For  example,  detection  of  the  onset  of  drowsiness  and 
fatigue  might  prevent  serious  problems  at  a  later  stage.  Similarly,  recognition  that  the  operator 
is  already  working  at  or  close  to  maximum  potential  serves  to  warn  that  the  imposition  of 
further  load  may  lead  to  impaired  performance.  A  number  of  systems  (e.g.,  Cognition  Monitor 
component  of  the  Cognitive  Cockpit)  have  been  designed  to  supply  the  type  of  information 
that  enables  these  inferences  to  be  made.  Critically,  they  have  been  designed  to  perform  these 
functions  in  real  time. 

6.2  General  Characteristics  of  Operator-State  Monitoring 
Systems 

Operator  state  is  a  general  term  that  is  used  to  characterise  the  overall  condition  of  an  operator 
at  any  particular  time.  It  refers  to  a  combination  of  behavioural  activity,  physiological  patterns 
and  subjective  states,  and  is  strongly  context-dependent.  In  considering  the  operator  as  an 
agent  within  a  complex  system,  the  term  ‘operator  state’  is  preferable  to  more  specific 
concepts  such  as  workload  (Pleydell-Pearce,  1999).  The  use  of  operator  state  is  preferred 
lapses  of  attention,  losses  of  awareness,  or  the  production  of  errors  are  not  necessarily 
restricted  to  periods  of  high  workload.  Thus,  in  attempting  to  characterise  operator  state, 
measurement  of  workload  is  only  one  of  the  functions  of  operator  state  assessment 
technologies. 

Inferences  about  operator  state  can  be  derived  from  four  principal  sources,  either  used 
individually  or  in  combination:  behavioural  measures;  physiological  measures;  subjective 
measures;  and  through  a  consideration  of  contextual  information.  Overall,  this  feedback 
provides  information  about  the  objective  and  subjective  state  of  an  operator  within  a  mission 
context.  This  information  then  provides  a  basis  for  the  intelligent  adaptation  of  the  interface  to 
best  support  the  operator. 

Sections  1 1.3  through  1 1.5  review  the  technologies  for  designing  behaviour-based  and 
physiological  based  interface  systems,  and  compare  the  differences  between  behaviour-based 
and  physiological-based  techniques  and  the  benefits  of  combining  the  two  techniques. 
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Reference: 


Young.  P.M.,  Clegg,  B.A.,  and  Smith,  C.A.P.  (2004).  Dynamics  models  of  augmented  cognition. 
International  Journal  of  Human-Computer  Interaction,  17(2),  259-273. 


Overview: 

This  paper  discusses  modeling  based  on  an  engineering  control  systems  theory  and  offers  insights 
into  closed-loop  systems;  that  is,  real-time  cognitive  state  detection.  Controls  system  theory  deals 
with  the  fundamental  properties  of  systems  described  by  mathematical  models.  Note  that  these 
mathematical  models  were  out  of  the  scope  of  this  review;  therefore,  please  refer  to  the  article  for 
further  information  regarding  the  implementation  of  the  mathematical  models. 

A  closed-loop  dynamic  model  entails  taking  a  measurement  of  cognitive  workload  (e.g., 
neurological  and  physiological  measures)  and  using  that  to  adapt  the  operator’s  input.  This  model 
required  a  cognitive  workload  and  action  model. 

It  was  shown  that  dynamic  instability  (that  is,  reliable  input  to  the  operator)  can  result  from 
introducing  feedback  within  a  system.  That  is,  rapid  detection  of  a  cognitive  state  under  high 
workload  might  result  in  input  being  removed,  which  would  reduce  the  workload  and  hence, 
additional  information  is  provided  (e.g.,  cluttered  display)  which  would  again  result  in  high 
workload,  etc.  The  authors  provide  some  methods  that  can  be  applied  to  remove  such  instability 
and  optimise  performance. 


Conclusions  for  IASs: 

1 .  A  control  systems  model  incorporating  the  flow  of  human  information  processing  (Perception, 
Decide,  Respond)  can  be  used  to  determine  mathematical  operations  (performance  properties 
of  the  system). 

2.  A  closed-loop  approach  can  be  used  to  examine  the  application  of  sensors  to  measure 
cognitive  states  and  to  enhance  human  performance,  and  vice  versa,  even  before  they 
physically  exist. 

3.  It  is  important  to  understand  the  affects  of  feedback  on  a  closed-loop  system  (e.g.,  stability, 
tracking  and  performance,  noise  and  disturbance  rejection,  and  bandwidth). 

4.  It  is  important  to  systematically  examine  the  effects  of  optimized  human  performance 
measures  in  regards  to  the  stability  of  the  input. 


Reference: 


Prinzel,  L.J.,  Freeman,  F.G.,  Scerbo,  M.W.,  Mikulka,  P.J.  and  Pope,  A.T.  (1999).  A  Closed  Loop 
System  for  Examining  Psychophysiological  Measures  for  Adaptive  Task  Allocation.  The 
International  Journal  Of  Aviation  Psychology,  10(4),  393-410. 
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Overview: 


This  paper  discusses  a  study  that  examined  the  effectiveness  of  a  closed-loop  system  to  moderate 
an  operator’s  level  of  engagement  where  the  automation  was  driven  by  the  operator’s  own  EEG. 
The  study  also  examined  the  impact  of  different  task  loads  on  adaptive  task  allocation  and  system 
regulation  of  task  engagement  and  workload. 

The  use  of  physiological  measures  in  this  study  is  based  on  the  concept  that  an  optimal  state  of 
engagement  exists.  It  is  thought  that  changes  in  arousal  and  resource  capacity  are  controlled  by 
feedback  from  other  ongoing  activities.  For  instance,  an  increase  in  the  task  load  can  enhance 
arousal  and  decrease  resources. 

Pope  et  al  (1995)  developed  an  adaptive  system  that  uses  a  closed-loop  procedure  to  adjust  the 
mode  of  automation  based  on  changes  in  an  operator’s  EEG  patterns.  The  closed-loop  method 
was  developed  to  determine  optimal  task  allocation  using  an  EEG-based  index  of  engagement  or 
arousal.  The  system  uses  a  biocybernetic  loop  that  is  formed  by  changing  levels  of  automation  in 
response  to  changes  in  mental  workload  demands.  Thus,  an  inverse  relation  exists  between  the 
level  of  automation  in  the  tasks  and  the  level  of  operator  workload.  The  study  applied  this  closed- 
loop  system  to  moderate  the  operator’s  level  of  engagement  and  apply  automation. 

Study  results  showed  that  performance  in  the  experimental  group  was  significantly  improved 
compared  to  the  control  group. 


Conclusions  for  IASs: 

1 .  Results  from  this  study  suggest  that  the  closed-loop  system  can  facilitate  performance. 

2.  A  closed-loop  feedback  system  can  provide  a  method  for  regulating  operator  attention, 
arousal,  and  workload,  and  represents  a  method  for  the  use  of  psychophysiological  measures 
in  adaptive  automation  technology. 


6.3  Behavioural-based  Monitoring 

6.3.1  Behavioural  Measures 

Measurement  of  the  overt  physical  behaviour  of  the  operator  can  provide  a  rich  database  that 
can  form  the  basis  for  inferences  concerning  the  current  state  of  the  operator.  For  example,  if 
an  operator  is  engaged  in  a  conversation  with  another  party,  this  indicates  that  the  verbal 
systems  within  the  operator’s  brain  will  be  heavily  committed  to  that  channel,  and  the  facility 
to  deal  with  or  act  upon  other  verbal  tasks  may  be  diminished.  This  inference  is  based  upon 
well-documented  problems  associated  with  parallel  processing  of  multiple  streams  of  verbal 
information.  Alternatively,  if  a  particular  system  is  currently  being  manipulated,  this  permits 
inferences  about  the  current  focus  of  attention,  and  provides  some  idea  about  current 
cognition  and  intention.  Furthermore,  sensors  that  track  head  and  eye  position  can  provide 
information  about  the  current  locus  of  visual  fixation  that  enables  inferences  about  the  locus 
of  visual  attention.  Combinations  of  such  behavioural  measures  can  provide  information  about 
time-sharing  and  dual-task  performance.  Finally,  inferences  can  be  made  about  operator 
cognitive  state  using  a  pre-existing  knowledge  base  in  which  the  functional  significance,  and 
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cognitive  implications,  of  system  controls  are  represented.  In  this  way,  behavioural 
monitoring  is  able  to  provide  some  degree  of  inferences  about  operator  intent. 

6.3.2  Subjective  Measures 

Subjective  measures  of  operator  state  are  those  provided  by  the  operator.  In  conventional 
settings  these  are  often  paper  and  pencil  tasks  (e.g.,  NASA  TLX).  However,  the  collection  of 
such  data  can  be  easily  automated;  for  example,  there  could  be  provision  for  the  operator  to 
signal  subjective  states  (i.e.,  the  operator’s  perception  of  their  current  level  of  workload)  that 
may  be  of  concern.  For  example,  states  of  very  high  workload  where  performance  may  be 
deteriorating  can  be  signalled  using  speech  or  a  single  button  press.  Similarly,  the  recognition 
of  chronic  under-arousal,  and  the  realisation  that  sleep  onset  can  also  be  communicated. 
Under  conditions  of  more-manageable  workload,  the  operator  can  also  signal  task  elements 
that  are  experienced  as  overly  demanding.  These  forms  of  subjective  data  are  invaluable 
indices  of  operator  state.  Incorporating  such  measures  within  a  closed-loop  system  directly 
links  the  operator  with  the  on-board  monitoring  systems,  and  as  a  result,  keeps  the  operator 
‘in-the-loop’. 

6.3.3  Contextual  Measures 

Context  provides  information  that  enables  interpretation  of  changes  in  behavioural, 
physiological  and  subjective  measures.  For  example,  aircraft  take-off  and  landing  are 
associated  with  dramatic  changes  in  physiological  variables  such  as  heart  rate. 

Inferences  made  about  operator  state  can  therefore  be  enhanced  by  information  about  general 
contextual  factors.  Furthermore,  since  effects  of  context  on  performance  may  be  predictable, 
this  allows  inferences  about  the  impact  of  that  context  on  overall  operator  state.  In  order  to 
achieve  this,  the  operator  state  monitoring  system  must  collect  low-level  contextual 
information  (e.g.,  ambient  noise,  luminance  and  temperature,  which  are  all  factors  known  to 
influence  performance). 


Wood,  S.  (2006).  Automated  behavior-based  interaction  customization  for  military  command  and 
control.  Technical  Report. 


Overview: 

This  paper  proposes  an  intelligent  control  framework  (ICF),  for  behaviour-based  customization  of 
an  adaptable  system.  The  authors  propose  that  this  framework,  while  developed  to  automate  and 
assist  with  warfighter  tasks,  can  also  be  applied  generically  to  any  adaptive  interface. 

Adjustable  Autonomy  Module.  This  module  is  a  component  of  IFC  that  dynamically  allocates  tasks 
by  using  a  combination  of  behavior  recognition,  modeling,  and  reasoning  tools.  The  module 
performs  the  following  operations: 

1 )  Senses  external  and  operator  stimuli; 

2)  Recognizes  new  tasks  and  monitors  the  progression  of  existing  tasks; 

3)  Simulates  operator  actions  to  determine  information  and  other  task  needs; 

4)  Updates  the  operator  and  situation  models; _ 
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5  )  Determines  automation  levels,  and, 

6)  Updates  the  instructions  for  control  over  automation. 

Intelligent  Control  Framework.  Inputs  from  the  operator,  system  status  information,  and  external 
events  are  used  as  inputs  to  the  system.  As  output,  the  system  produces  a  model  that  includes  the 
operator’s  current  situation,  and  a  table  of  adjustable  autonomy  and  data  on  the  current  tasks  with 
associated  autonomy  levels.  The  autonomy  levels  indicate  which  tasks  the  system  should  be 
engaged  in  at  any  point  in  time,  to  what  degree  the  system  hould  be  performing  or  monitoring 
those  tasks,  and  links  the  tasks  back  to  information  in  the  user  model. 

Conclusions  for  IASs: 

1 .  Design  of  the  adjustable  autonomy  module  architecture  raised  several  scientific  and 
engineering  issues,  which  are  issues  critical  within  any  application  of  behavior-based 
customization: 

a.  How  does  behavior-based  customization  fit  within  a  larger  control  hierarchy/system? 

b.  Since  operators  rarely  perform  one  action  at  a  time,  how  best  can  individual  operator 
actions  be  interpreted  and  assessed  with  regard  to  current  or  new  tasks?  How  can  task 
progress  be  accurately  measured?  How  can  an  operator  deal  with  canceled, 
intermittent,  or  suspended  tasks? 

c.  How  must  these  traditional  modeling  techniques  be  augmented  to  adequately  deal  with 
expected  world  events,  task  priorities,  operator  information  needs,  etc.,  and  use  that 
additional  information  to  more  accurately  recognize  operator  action  plans? 

d.  How  must  these  underlying  technologies  be  redesigned  to  enable  smooth  transition 
from  operator  control  to  system  control  without  creating  an  undue  mental  context¬ 
switching  cost  on  the  operator? 

e.  How  will  operators  interact  with  the  system  for  maximal  benefit? 


Alpert,  S.  R.,  Karat,  J.,  Karat,  C.  M.,  Brodie  C.,  &  Vergo,  J.  G.  (2003).  User  attitudes  regarding  a 
user-adaptive  e-commerce  web  site.  User  Modeling  and  User-Adapted  Interaction  13(4),  pp.  373- 
396. 


Overview: 

A  study  to  evaluate  operator  attitudes  towards  an  adaptive  Ul  for  an  eCommerce  Web  site  based 
on  explicit  and  implicit  operator  model  was  conducted.  The  implicit  user  model  (behaviour)  is 
based  on  previous  navigation  with  the  interface.  The  explicit  model  is  based  on  direct  feedback 
from  the  operator. 


Conclusions  for  IASs: 

1 .  Results  show  that  operators  wanted  explicit  control  of  their  operator  profiles  due  to  trust 
issues.  Wanting  this  sense  of  control  was  partially  related  to  whether  the  participant  could 
readily  make  sense  of  the  interaction  with  a  web  site  (i.e.,  have  a  proper  mental  model  of 
system  functioning). 

2.  Participants  reported  that  they  were  happy  with  adaptive  content  based  on  information 
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explicitly  provided  by  them,  but  were  mixed  in  their  reactions  to  adaptivity,  based  on  implicit 
information. 

3.  Operators  expected  the  system  to  respond  to  the  ongoing  dialog,  and  to  be  as  informative  as 
only  required. 

4.  A  highly  rated  feature  was  “content  filtering”  based  on  explicit  information  provided  by  the 
operator  or  explicitly  selected  by  the  operator,  or  information  which  is  explicitly  accessible  in 
the  operator’s  profile. 

5.  An  explicit  user  model  can  help  operators  develop  a  mental  model  of  the  system;  that  is, 
operators  obviously  understand  the  causal  relationships  that  give  rise  to  the  content.  When 
immediate  context  is  used  to  guide  content,  operators  can  easily  infer  the  source  of  the  site’s 
content.  Workload  increases  however  with  an  explicit  user  model. 


Reference: 


Bonner,  M.C.,  Taylor,  R.  M.,  Fletcher,  K.,  and  Miller,  C.  (2000).  Adaptive  automation  and  decision 
aiding  in  the  military  fast-jet  domain.  In  proceedings  of  the  conference  on  Human  Performance, 
Situation  Awareness  and  Automation:  User  centred  design  for  the  new  Millennium. 


Overview: 

This  paper  presents  the  operation  and  technical  development  of  the  Tasking  Interface  Manager 
component  of  the  Cognitive  Cockpit.  The  TIM  utilised  input  from  the  Situation  Assessment  Support 
System  and  the  Cognition  Monitor  to  adaptively  present  information  and  adaptively  automate  tasks 
according  to  the  situational  context  and  a  pilot's  internal  state.  The  goal  of  TIM  is  to  reduce  aircrew 
task  and  cognitive  load.  The  main  feature  of  the  TIM  is  a  shared  mental  model,  the  ability  to  track 
goals,  plans  and  tasks,  and  the  ability  to  communicate  intent  about  the  mission  plan.  The  objective 
of  the  TIM  is  to  allow  aircrew  to  retain  executive  control  of  aircraft  and  mission  parameters  in 
conjunction  with  the  assistance  of  adaptive  automation. 


Conclusions  for  IASs: 

1 .  To  maintain  operator  situational  awareness,  tasks  should  be  tracked  explicitly  (e.g.,  by  asking 
the  operator  for  input  or  by  making  the  system  state  visible  to  the  operator),  especially  in  high- 
criticality  environments. 


Reference: 


Gerlach,  M.  and  Onken,  R.  (1995).  CASSY  -  The  electronic  part  of  the  human-electronic  crew. 
Proceedings  of  the  3rd  international  workshop  on  human-computer  teamwork  (Human-Electronic 
Crew:  Can  we  trust  the  team?).  Cambridge,  UK,  27-30  September  1994. 


Overview: 

The  knowledge-based  commercial  aircraft  Cockpit  Assistant  System  is  a  civil  aviation  cockpit 
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assistant  project  developed  as  an  intelligent  decision  aid.  It  emphasises  pilot  assistance  through 
situation  assessment  and  re-planning  in  flight.  Situation-dependent  assistance  with  flight  planning 
is  guided  by  a  normative  pilot  model,  goal  conflict,  pilot  intent,  and  error  recognition  functions.  It 
also  aids  in  the  execution  of  pilot  selected  functions. 

CASSY  is  composed  of  several  situation  assessment  modules  that  interface  with  the  flight  crew, 
the  aircraft,  and  air  traffic  control,  which  all  collaborate  with  each  other  The  CASSY  project  is  a 
successful  real-time  demonstration  of  an  intelligent  adaptive  system  implemented  in  a  real  and  not 
virtual  environment.  This  project  led  to  the  CAMMA  military  cockpit  assistant  project. 


Conclusions  for  IASs: 

1 .  Flight  tests  proved  that  intelligent  decision  aiding  is  feasibly  possible,  and  well  accepted  by 
operators. 

2.  Situation  assessment  is  an  important  feature  of  a  successful  intelligent  system. 


Wittig,  T.  and  Onken,  R.  (1992).  Pilot  intent  and  error  recognition  as  part  of  a  knowledge  based 
cockpit  assistant.  Proceedings  of  the  AGARD  conference  on  Combat  automation  for  airborne 
weapon  systems:  Man/machine  interface  trends  and  technologies,  Edinburgh,  UK,  19-22  October 
1992. 


Overview: 

This  paper  describes  the  concept  and  functionality  of  the  Cockpit  Assistant  System  (CASSY); 
including  pilot  intent  and  error  recognition.  Evaluation  of  CASSY  in  a  flight  simulator  is  also 
described. 


Conclusions  for  IASs: 

1 .  Operator  intent  and  error  recognition  can  be  an  effective  means  of  providing  adaptive 
assistance. 

2.  Uncertainties  can  be  evaluated  using  certainty  factors  (probabilistic  reasoning  such  as  Bayes’ 
Theorem). 

3.  Algorithms  based  on  a-priori  probabilities  for  possible  hypotheses  have  proven  useful  for 
recognizing  and  estimating  operator  intent.  The  probabilities  can  be  modified  with  respect  to 
operator  actions. 


6.4  Psychophysiological-based  Monitoring 

6.4.1  Physiological  Measures 

Although  behavioural  measures  are  clearly  useful,  they  by  no  means  provide  a  full  and 
definitive  picture.  For  example,  a  response  in  itself  may  provide  no  information  about  current 
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levels  of  arousal  and  alertness.  Furthermore,  research  over  the  last  decade  has  indicated  that 
changes  in  cognitive  load  have  predictable  effects  upon  physiological  measures,  particularly 
those  occurring  within  the  brain.  This  means  that  physiological  measures  can  provide  an 
objective  and  non-invasive  index  of  load  imposed  upon  distinct  brain  systems  that  have 
specific  functions.  A  number  of  physiological  variables  that  can  provide  such  an  index  of 
cognitive  load  are  described  in  Sections  1 1 .4. 1 . 1  through  1 1 .4. 1 .8. 

6.4.  1 . 1  Electroencephalographic  Measures 

Sensors  placed  in  contact  with  the  scalp  are  able  to  detect  electrical  changes  within  the  brain. 
Although  these  voltages  are  small,  they  can  be  measured  when  the  signals  are  passed  through 
high-gain  amplifiers.  These  measures  of  EEG  activity  are  closely  associated  with  behavioural 
state  (e.g.,  under  and  over-arousal,  high-order  cognition,  modality-specific  processing,  verbal 
and  spatial  processing,  etc.).  Most  measures  of  EEG  activity  are  in  a  bandwidth  lying  between 
0  and  400Hz  at  widespread  regions  across  the  scalp.  They  are  able  to  make  inferences  about 
activity  in  functionally  specific  regions  of  the  brain  (e.g.,  visual,  auditory,  somatosensory  and 
frontal  executive  regions),  and  also  provide  an  index  of  both  drowsiness  and  alertness  using 
changes  in  the  power  of  particular  frequencies  (e.g.,  delta,  theta,  alpha,  beta  and  gamma).  A 
problem  with  interpreting  EEG,  stems  from  the  fact  that  individual  brains  may  differ  in 
organisation,  and  that  different  strategies  and  styles  are  associated  with  distinct  patterns  of 
EEG  activity.  Another  problem  associated  with  EEG  recording  is  that  eye-movements  and 
blinks  produce  far-field  potentials  which  are  detected  by  scalp  electrodes. 

6.4.1 .1 .1  Electro-oculographic  (EOG)  Measures 

Electrodes  placed  above  and  below  an  eye  are  able  to  detect  eye-movements,  and  signal  the 
occurrence  of  blinks.  These  sensors  do  not  impede  normal  vision.  EOG  measures  of  eye- 
movements  can  provide  extremely  accurate  measures  of  fixation  under  ideal  conditions.  Blink 
rate  has  been  demonstrated  to  correlate  with  visual  workload;  blink  rate  reduces  when  visual 
workload  increases.  Increased  blink  frequency,  and  longer  duration  blinks  have  also  been 
related  to  fatigue  and  the  onset  of  sleep.  Saccade  rate  provides  an  index  of  visual  scan  rate, 
and  provides  an  approximate  measure  of  visual  shifts. 

6.4. 1.2  Electrodermal  Activity 

Electrodermal  activity  at  the  skin  surface  has  been  used  as  a  measure  of  autonomic  activity  for 
many  decades.  Although  numerous  systems  have  been  used,  most  measure  changes  in  skin 
impedance/resistance.  Electrodermal  activity  is  measured  using  a  small  sensor  array  attached 
to  the  skin.  Changes  primarily  arise  as  a  result  of  alterations  in  sweat  gland  activity;  increased 
sweat  gland  activity  indicates  increased  autonomic  arousal.  While  electrodermal  changes  do 
not  provide  a  direct  measure  of  higher  level  cognition,  they  do  indicate  increments  and 
reductions  of  arousal  and  stress  reactions.  This  is  important,  as  research  over  several  decades 
has  shown  that  excessive  increments  in  autonomic  arousal  are  associated  with  dysfunctional 
attentional  narrowing,  distraction  by  irrelevant  inputs,  and  marked  impairments  in  cognitive 
task  performance. 
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6.4. 1.3  Heart  Rate  and  Heart  Rate  Variability 

Increments  in  load  have  been  shown  to  be  accompanied  by  systematic  changes  in  cardiac 
activity.  A  simple  measure  of  cardiac  activity  is  supplied  by  heart  rate,  expressed  in  beats  per 
minute.  Increments  in  workload  have  been  associated  with  increased  heart  rate.  Inter-beat 
interval  has  also  been  found  to  correlate  with  changes  in  workload.  These  measures  are 
derived  from  ECG  electrodes  placed  in  contact  with  the  skin. 

6.4.1. 4  Respiration  Measures 

Measures  of  respiration  (e.g.,  breathing  rate)  are  indicative  of  operator  state;  shallower  and 
higher  frequency  patterns  of  respiration  have  been  shown  to  be  associated  with  increments  in 
stress  and  cognitive  demand,  and  respiration  rate  exerts  significant  effects  upon  heart  rate  and 
heart  rate  variability.  Thus,  frequency  decomposition  of  the  respiratory  cycle  enables  an 
assessment  of  the  degree  to  which  cardiac  changes  are  artefacts  of  respiratory  variability. 
Respiration  rate  can  be  measured  using  a  strain  gauge  or  the  “dolls  eye”  located  within 
aircraft  respiratory  systems. 

6.4.1. 5  Skin  Temperature  Measures 

Variations  in  skin  temperature  have  also  been  related  to  changes  in  autonomic  activity; 
increased  activity  in  the  sympathetic  nervous  system  results  in  vasoconstriction  of  peripheral 
arteries  which  lowers  skin  temperature  at  bodily  extremities.  For  this  reason,  decrements  in 
peripheral  temperature  have  been  used  as  measures  of  stress  and  arousal. 

6.4. 1. 6  Electromyographic  Measures  (EMG) 

Electrodes  attached  to  skin  are  sensitive  to  activity  in  underlying  muscle  groups.  Muscle 
activity  is  predominantly  associated  with  frequencies  in  the  range  of  10Hz  and  upwards  and  is 
particularly  marked  between  50Hz  and  150Hz.  Such  measures  are  indicative  of  effector 
workload  and  more  physically  onerous  acts  are  associated  with  higher  amplitude  EMG 
activity.  This  means  that  alterations  in  peripheral  load  associated  with  control  of  devices,  such 
as  joysticks,  can  be  mapped  in  real  time.  EMG  measures  also  correlate  with  state  variables 
such  as  drowsiness.  Indeed  during  drowsiness,  sleep  onset  and  sleep  itself,  there  is  a 
progressive  decrease  in  muscle  activity.  Thus  a  decrease  in  muscle  tonus  may  indicate 
dangerously  low  levels  of  alertness.  In  contrast,  states  of  higher  arousal  are  associated  with 
increasing  muscle  tonus. 

6.4.1. 7  Vocalisation  and  Auditory  Communication  Detection  Systems 

A  major  source  of  workload  stems  from  the  requirement  of  operators  to  use  verbal 
communication.  This  includes  using  speech  recognition  software  to  interact  with  the  system 
and  communication  with  other  operators  and  remote  stations.  There  are  two  techniques  to 
measure  vocalisation  and  auditory  communication;  the  first  technique  measures  vocalisation 
via  electrodes  attached  at  the  skin  surface  around  the  larynx,  and  the  second  technique  uses 
analog  to  digital  conversion  and  frequency  analysis  of  information  passing  through  pilot 
microphones.  The  second  technique  is  advantageous  as  it  is  simple  and  non-invasive.  When 
the  system  detects  vocalisation,  it  infers  that  the  operator  is  paying  attention  to  verbal 
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information,  indicating  that  additional  verbal  load  in  any  modality  is  likely  to  be  processed 
less  effectively. 

6.4. 1.8  Ambient  Sensors 

Operator  state  monitoring  systems  also  make  use  of  a  number  of  sensors  providing 
information  about  ambient  factors.  These  include  ambient  noise,  since  increments  in  noise  are 
known  to  influence  performance,  and  ambient  temperature  and  luminance,  since  the  extremes 
of  both  these  measures  have  well  documented  and  deleterious  effects  on  performance. 


Parasuraman,  R.  (2003).  Neuroergonomics:  research  and  practice.  Theoretical  Issues  in 
Ergonomics.  Science,  4  (1-2),  5-20. 


Overview: 

This  article  describes  the  characteristics  and  scope  of  neuroergonomics.  This  is  defined  as  the 
study  of  brain  and  behaviour. 

The  author  details  the  advantages  and  disadvantages  of  Neuroergonomics.  Neuroergonomics 
investigates  the  neural  bases  of  various  perceptual  and  cognitive  functions  (such  as  seeing, 
hearing,  attending,  remembering,  deciding  and  planning)  in  relation  to  technologies  and  settings  in 
the  real  world.  The  basic  principle  of  neuroergonomics  is  to  understand  how  the  brain  works  to 
perform  various  tasks.  A  core  feature  of  neuroergonomics  is  an  interest  in  brain  mechanisms  in 
relation  to  human  performance  at  work. 

Neuroergonomics  has  two  major  goals  when  looking  for  links  between  brain  function  and  the  world 
of  technology  and  work.  The  first  is  to  design  technologies  by  using  existing  and  emerging 
knowledge  of  human  performance  and  brain  function.  The  second  goal  is  to  enhance  our 
understanding  of  brain  function  in  relation  to  human  performance  in  real-world  tasks. 

Neuroergonomic  measures  offer  new  ways  to  understand  how  to  implement  adaptation.  For 
instance,  changes  in  reaction  time  may  reflect  contributions  of  both  central  processing  (working 
memory)  and  response-related  processing  to  workload.  However,  when  coupled  with  the  amplitude 
and  latency  of  the  P300  component  of  the  ERP,  such  changes  may  be  more  precisely  localized  to 
central  processing  stages  than  to  response-related  processing.  In  addition,  measures  of  brain 
function  can  indicate  not  only  when  an  operator  is  overloaded,  drowsy,  or  fatigued,  but  also  which 
brain  networks  and  circuits  may  be  affected. 

Below  outlines  the  benefits  and  costs  of  various  neurophysiological  measures  for  adaptive 
automation  and  understanding  of  brain  function  in  relation  to  human  performance. 


Conclusions  for  IASs: 

1 .  Brain  imaging  techniques.  Offer  the  most  direct  measure  of  brain  functioning.  Examples 
include  EEG,  MEG,  ERPs,  MRI,  fMRI,  TMS.  These  techniques  can  be  expensive  and  restrict 
movement  but  new  technologies  are  becoming  cheaper  and  more  portable. 

2.  PET  and  fMRI  measures  of  brain  activity,  as  well  as  electromagnetic  measures  such  as  EEG 
and  ERPs,  provide  sensitive  indexes  of  moment-to-moment  variations  in  mental  workload  in 
adaptive  human-machine  systems. 

3.  Transcranial  Doppler  sonography  (TCP)  has  high  temporal  resolution,  thus  allowing  for 
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continuous,  real-time  monitoring  of  cerebral  blood  flow. 

4.  Physiological  measures  recorded  from  the  body  (e.g.,  heart  rate,  skin  conductance,  urinary 
catecholamines,  blood  pressure,  etc.)  focus  on  autonomic  nervous  system  (ANS)  measures  in 
relation  to  somatic  factors,  emotion  and  stress. 

5.  Neuroergonomics  and  psychophysiology  in  ergonomics  share  a  common  goal  of  seeking  the 
design  of  safe  and  efficient  human-machine  systems.  The  two  can  be  considered 
complementary  and  overlapping  approaches. 

6.  Physiological  measures  may  be  recorded  continuously  without  overt  responses  and  may 
provide  a  measure  of  the  covert  activities  of  the  human  operator. 

7.  In  some  instances,  measures  of  brain  function  may  provide  more  information  when  coupled 
with  behavioural  measures  than  using  behavioural  measures  alone. 

8.  Physiological  measures  can  possibly  predict  human  error  by  analysis  of  brain  activity  that  has 
been  previously  associated  with  errors.  For  instance,  a  specific  ERP  component  associated 
with  errors  has  been  identified,  the  error-related  negativity  (ERN).  Errors  made  in  a  choice 
reaction  time  task  in  which  either  the  hand  or  the  foot  was  used  to  respond  led  to  nearly 
identical  ERN. 

9.  Analysis  of  learning  a  complex  task  can  be  completed  by  understanding  the  brain  changes  that 
accompany  stages  of  learning.  This  could  lead  to  the  development  of  better  training 
procedures.  PET  and  fMRIs  can  be  used  for  short  and  long  term  studies  of  learning. 

10.  Other  applications,  brain-machine  interface  (controlling  external  devices  with  brain  potentials) 
and  understanding  mechanisms  of  spatial  navigation  could  have  important  implications  for 
further  understanding  of  the  mechanisms  of  spatial  navigation  and  its  acquisition  in  expert 
groups,  such  as  pilots  and  controllers. 


Russell,  C.A.  (2005).  Operator  State  Estimation  for  Adaptive  Aiding  in  Uninhabited  Combat  Air 
Vehicles,  Dissertation  (Report  No.  AFIT/DS/ENG/05-01).  Air  Force  Institute  Of  Technology,  Wright- 
Patterson  Air  Force  Base,  Ohio. 


Overview: 

This  document  is  a  dissertation  that  describes  a  series  of  experiments  to  examine  the 
effectiveness  of  a  closed-loop  system,  based  on  an  operator’s  cognitive  functional  state,  to 
adaptively  aid  in  UCAV  (Uninhabited  Combat  Air  Vehicle)  operations.  Specifically,  it  examined  how 
this  closed-loop  system  can  help  deal  with  high  workload  situations  without  disengaging  the 
operator  from  the  task. 

Adaptive  aiding  was  implemented  based  on  “operator  state  estimation”  whereby  the  system  adapts 
when  the  operator  is  cognitively  loaded. 

The  operator  functional  state  was  determined  by  integrating  and  assessing  multiple 
psychophysiological  measures  using  an  operator  state  classification  system.  That  system  was  then 
used  to  change  the  environment. 

Operator  state  has  four  major  components:  psychophysiological  assessment  (cognitive  workload); 
operator  performance  assessment;  situation  awareness  assessment;  and  momentary  mission 
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requirements.  The  primary  focus  in  this  research  is  closing  the  loop’  of  the  human-machine  system 
based  on  cognitive  workload  alone. 

The  author  reports  on  studies  which  suggest  that  EEG  measures  can  be  used  to  determine 
multiple  levels  of  cognitive  load  in  complex  tasks.  EEG  measures  are  sensitive  to  cognitive 
differences  and  reliable  enough  for  consistent  use.  Artificial  neural  networks  meanwhile  have  been 
used  in  both  simple  single-task  laboratory  and  complex  multiple-task  studies  to  classify  cognitive 
workload. 

The  integration  of  system  adaptive  automation  and  natural  human  adaptation  must  be 
accomplished  to  eliminate  the  possibility  of  human-system  instability;  operators  themselves  are 
adaptable  and  can  respond  to  systems  in  unpredictable  ways.  This  integration  may  be 
accomplished  by  adding  psychophysiological  measures  to  the  existing  system. 


Conclusions  for  IASs: 

1 .  Artificial  neural  networks  (ANNs)  have  been  successfully  used  to  accurately  classify  cognitive 
workload  (differences  in  EEG)  in  a  variety  of  environments. 

2.  Adaptive  aiding  based  on  psychophysiological  measures  (i.e.,  EEG)  can  improve  operator 
performance  and  increase  mission  effectiveness.  This  effectiveness  however  is  dependent  on 
providing  aid  at  appropriate  times.  In  this  study,  operators  who  were  aided  at  random  times 
had  the  same  performance  as  unaided  operators. 

3.  Result  found  that  adaptive  aiding  through  a  closed-loop  system  improved  operator 
performance  and  increased  mission  effectiveness  by  67%. 


Wilson,  G.F.  (2002).  Adaptive  Aiding  Implemented  by  Psychophysiologically  Determined  Operator 
Functional  State.  In  Proceedings  of  RTO  Human  Factors  and  Medicine  Panel  (HFM)  Symposium 
held  in  Warsaw,  Poland,  7-9  October  2002. 


Overview: 

This  paper  provides  examples  for  the  application  of  psychophysiological  measures  (mental 
workload,  fatigue,  and  inattention)  to  classify  operator  functional  state  (OFS). 

The  author  reports  on  studies  which  evaluated  various  psychophysiolocial  measures  for  adapting 
automation.  These  studies  demonstrate  that  psychophysiological  measures  can  provide  high 
levels  of  accuracy  in  predicting  changes  in  operator  state.  Correct  classification  of  OFS  accuracies 
range  from  85%  to  98%. 

An  example  of  this  approach  is  using  a  stepwise  discriminant  analysis  and  artificial  neural  network 
classifiers  to  determine  how  well  they  could  correctly  classify  the  mental  workload  of  operators. 


Conclusions  for  IASs: 

1 .  Psychophysiological  measures  can  provide  a  more  accurate  estimate  of  OFS  status  to  adapt 
the  interface  in  real-time  minimal  interference. 

2.  The  classification  of  OFS  must  be  highly  accurate  in  order  to  provide  useful  information. 


222 


3.  The  situation  awareness  (SA)  of  operators  should  be  evaluated  in  light  of  the  current  mission 
requirements.  The  psychophysiological  and  SA  evaluations  could  then  be  used  to  determine 
their  functional  state.  The  current  mission  requirements  would  provide  the  context  with  which 
to  determine  if  the  current  OFS  was  appropriate  or  if  adaptive  aiding  was  required. 


Reference: 


St  John,  M.,  Kobus,  D.A.,  and  Morrison,  J.G.  (2003).  DARPA  Augmented  Cognition  Technical 
Integration  Experiment  (TIE).  DARPA  Technical  Report  1905.  December  2003. 


Overview: 

This  paper  describes  the  empirical  results  of  a  Technical  Integration  Experiment  (TIE)  involving  the 
evaluation  of  20  psychophysiologically  derived  measures  (cognitive  state  gauges)  that  were 
developed  under  Phase  I  of  the  Augmented  Cognition  program.  A  key  attribute  of  the  TIE  was  the 
use  of  a  common  experimental  test  task,  evaluated  under  comparable  test  conditions.  This  report 
attempts  to  examine  the  prospects  for,  and  issues  that  must  yet  be  addressed  for,  the  successful 
transition  of  these  cognitive  state  gauges  to  field-able  military  person-machine  systems  in  Phase  II 
of  the  Augmented  Cognition  program,  and  beyond. 

The  sensor  technologies  included  functional  Near  Infra-Red  imaging  (fNIR),  continuous  and  event- 
related  electrical  encephalography  (EEG/ERP),  eye  tracking  and  pupil  dilation,  mouse  pressure, 
body  posture,  heart  rate,  and  galvanic  skin  response  (GSR). 

Refer  to  the  report  for  details  on  the  advantages  and  disadvantages  of  each  gauge. 


Conclusions  for  IASs: 

1 .  When  implementing  psychophysiological  gauges,  there  is  a  need  to  define  and  refine  the 
meaning  of  each  gauge  and  what  it  measures. 

2.  Level  of  operator  experience.  When  evaluating  a  gauge,  the  system  design  needs  to 
understand  how  operator  skills  and  experience  (including  practice  effects)  can  influence  gauge 
performance. 

3.  Operator  acceptance.  To  maximize  the  likelihood  of  operator  acceptance,  gauge  hardware 
should  be  comfortable,  mobile,  convenient  and  as  non-intrusive  as  possible,  particularly  in 
mobile  environments. 

4.  Potential  sources  of  electro-magnetic  frequency  (EMF)  interference  need  to  be  understood  and 
addressed. 


Reference: 


Prinzel,  L.J.,  Freeman,  F.G.,  Scerbo,  M.W.,  Mikulka,  P.J.  and  Pope,  A.T.  (1999).  A  Closed  Loop 
System  for  Examining  Psychophysiological  Measures  for  Adaptive  Task  Allocation.  The 
International  Journal  Of  Aviation  Psychology,  10(4),  393-410. 
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Overview: 


This  paper  discusses  a  study  that  examined  the  effectiveness  of  a  closed-loop  system  to  moderate 
an  operator’s  cognitive  level  of  engagement  in  the  task,  where  the  automation  was  driven  by  the 
operator’s  own  EEG.  The  study  also  examined  how  different  task  loads  impact  adaptive  task 
allocation  and  system  regulation  of  task  engagement  and  workload. 

Physiological  measures  were  used  in  this  study  based  on  the  idea  that  an  optimal  state  of 
cognitive  engagement  in  the  task  exists.  Changes  in  arousal  and  resource  capacity  are  thought  to 
be  controlled  by  feedback  from  other  ongoing  activities.  For  instance,  an  increase  in  the  task  load 
for  activities  can  enhance  arousal  and  decrease  resources. 

Pope  et.  Al.,  (1995)  developed  an  adaptive  system  that  uses  a  closed-loop  procedure  to  adjust  the 
mode  of  automation  based  on  changes  in  the  operator’s  EEG  patterns.  The  closed-loop  method 
was  developed  to  determine  optimal  task  allocation  using  an  EEG-based  index  of  engagement  or 
arousal.  The  system  uses  a  biocybernetic  loop  that  is  formed  by  changing  levels  of  automation  in 
response  to  changes  in  mental  workload  demands.  Thus,  an  inverse  relation  exists  between  the 
level  of  automation  in  the  tasks  and  the  level  of  operator  workload.  The  current  study  applied  this 
closed-loop  system  to  moderate  the  operator’s  level  of  engagement  and  apply  automation. 

Study  results  indicate  that  performance  in  the  experimental  group  was  significantly  improved 
compared  to  the  control  group. 


Conclusions  for  IASs: 

1 .  The  authors  report  that  an  advantage  of  biopsychometrics  for  adaptive  systems  is  that 
measurements  can  be  obtained  continuously  with  little  intrusion.  Also,  since  obvious 
performance  indices  are  difficult  to  achieve  when  operators  interact  with  systems,  there  may 
be  fewer  opportunities  to  measure  resource  capacity. 

2.  Task  allocation  and  psychophysiological  data  can  complement  each  other  to  support  adaptive 
task  allocation. 

3.  Technology  is  still  immature  to  make  psychophysiological  measures  applicable  outside  the 
laboratory. 


Reference: 


Miller,  C.A.,  and  Dorneich,  M.C.  (2006).  From  Associate  Systems  to  Augmented  Cognition  25 
Years  of  User  Adaptation  in  High  Criticality  Systems.  Poster  presented  at  the  Augmented 
Cognition  conference,  October  2006,  San  Francisco. 


Overview: 

In  the  1980’s,  the  U.S.  Air  Force  initiated  the  development  of  a  human-adaptive,  information,  and 
automation  management  technology  known  as  the  “Pilot’s  Associate”. 

PA,  and  all  of  the  subsequent  associate  systems,  consisted  of  an  integrated  suite  of  intelligent 
subsystems  that  were  designed  to  share  (among  themselves  and  with  the  pilot)  a  common 
understanding  of  the  mission,  the  current  state  of  the  world,  the  aircraft  and  the  pilot.  Associate 
systems  were  designed  to  use  the  shared  knowledge  to  plan  and  suggest  courses  of  action,  and  to 
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adapt  cockpit  information  displays  and  the  behaviour  of  aircraft  automation. 
Please  refer  to  the  Frameworks  Section  8.4. 2. 2  for  more  details  on  this  paper. 


Conclusions  for  IASs: 

1 .  Importance  of  co-development  and  progressive  testing :  Development  efforts  and  individual 
technologies  should  be  co-developed  and  used  in  collaboration,  which  can  help  aid  in  the 
development  of  an  overall  system.  For  instance,  the  development  of  neurophysiological 
sensors  or  “meters”,  and  other  means  of  assessing  operator  state,  and  the  development  of 
methods  for  “augmenting”  cognition  through  information  display  technologies  need  to  be  co¬ 
developed  and  evaluated  in  concert. 

2.  Benefits  of  an  explicit,  integrative  framework  (task  model).  Knowledge  of  the  task  context  can 
help  develop  systems  that  manage  task  demand  and  increases  operator  performance. 

3.  Importance  of  learning,  especially  individuation.  Recording  individual  performance  effects 
could  serve  to  provide  a  powerful  means  of  adapting  system  behaviour  to  the  individual. 


Prendinger,  H.,  Ma,  C.,  Ishizuka,  M.  (2007).  Eye  movements  as  indices  for  the  utility  of  life-like 
interface  agents:  A  pilot  study.  Interacting  with  Computers,  19,  281-292. 


Overview: 

This  paper  outlines  a  pilot  study  that  compared  three  types  of  media:  an  animated  agent,  a  text 
box,  and  speech  only.  Authors  propose  a  different  approach  to  evaluating  animated  agents,  one 
that  is  based  on  eye  movement  behavior  of  operators  interacting  with  the  OMI.  The  operators  do 
not  manipulate  the  interface.  The  authors  argue  that  operators  merely  watching  a  presentation 
interact,  even  involuntarily,  with  their  eye  movements. 

Physiological  signals  as  an  evaluation  method  for  OMIs  and  as  an  input  modality:  The  eye 
movement  data  was  analyzed  for  diagnostic  use  (as  a  means  to  examine  the  operator’s  attention 
to  evaluate  the  usability  of  interfaces),  and  for  interactive  use  (a  system  responds  to  the  observed 
eye  movements  and  can  thus  be  seen  as  an  input  modality). 

The  investigation  of  eye  movements  revealed  that  deictic  gestures  by  the  agent  are  more  effective 
in  directing  the  attentional  focus  of  subjects  to  relevant  interface  objects  than  the  media  used  in  the 
two  control  conditions,  at  a  slight  cost  of  distracting  the  operator  from  visual  inspection  of  the 
object  of  reference.  The  results  also  demonstrate  that  the  presence  of  an  interface  agent 
seemingly  triggers  natural  and  social  interaction  protocols  of  human  operators. 


Conclusions  for  IASs: 

5.  Physiological  signals  can  be  used  as  an  evaluation  method  for  OMIs  and  as  an  input  modality. 

6.  Eye  movement  data  might  offer  valuable  information  relevant  to  the  utility  of  life-like  agents. 

7.  The  usability  of  interfaces  can  be  assessed  using  eye  movements. 

8.  The  tracking  of  eye  movements  can  capture  an  operator’s  interaction  with  the  system  in  real 
time,  which  is  hard  to  accomplish  using  post-experiment  questionnaires. 
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Prinzel,  LJ.,  Parasuraman,  R.,  Freeman,  F.G.,  Scerbo,  M,  Mikulka.,  P.J.,  and  Pope,  A.  (2003). 
Three  Experiments  Examining  the  Use  of  Electroencephalogram,  Event-Related  Potentials,  and 
Heart-Rate  Variability  for  Real-  Time  Human-Centered  Adaptive  Automation  Design.  Technical 
Report  (No.  NASA/TP-2003-21 2442). 


Overview: 

This  paper  describes  on  three  experiments  that  examined  the  psychophysiological  measures  of 
event-related  potentials,  electroencephalogram,  and  heart-rate  variability  for  real-time  adaptive 
automation. 

The  authors  base  their  closed-loop  system  on  a  theoretical  framework  proposed  by  Byrne  and 
Parasuraman  (1996).  This  framework  is  proposed  for  developing  adaptive  automation  around 
psychophysiological  measures.  The  use  of  physiological  measures  in  adaptive  systems  is  based 
on  the  assumption  that  there  exists  an  optimal  state  of  cognitive  engagement  in  a  particular  task. 
Capacity  and  resource  theories  are  central  to  this  idea.  These  theories  posit  that  a  limited  amount 
of  resources  exist  that  can  be  drawn  upon  when  performing  tasks.  These  resources  are  not  directly 
observable,  but  instead  are  hypothetical  constructs.  Therefore,  physiological  measures  can  be 
used  to  index  cognitive  resources. 

A  closed-loop  system  can  compliment  task  allocation.  The  operators  may  be  better  able  to  predict 
the  “state”  of  system  operation,  develop  control  strategies,  select  appropriate  actions,  and  interpret 
the  effects  of  selected  actions  with  appropriate  feedback. 

The  results  of  the  experiments  confirm  that  these  measures  can  be  an  effective  tool  for  use  in  both 
a  developmental  and  operational  role  for  adaptive  automation  design. 


Conclusions  for  IASs: 

1 .  The  use  of  psychophysiological  measures  in  adaptive  automation  requires  that  such  measures 
are  capable  of  representing  mental  workload. 

2.  It  has  been  proposed  that  physiological  measures  in  adaptive  systems  could  be  used  to  keep 
the  operator  in  an  “optimal  state  of  engagement”. 

3.  It  has  been  reliably  shown  that  event-related  potentials  accurately  measure  mental  workload. 
ERP  however  is  fairly  intrusive  and  difficult  to  implement  and  requires  considerable  expertise 
to  interpret  the  results. 

4.  Heart-rate  variability  (HRV)  is  not  as  intrusive  or  difficult  to  implement  as  ERP,  and  is  easy  to 
use  and  reliable.  However,  it  does  not  have  the  same  capability  as  the  ERP  in  terms  of  its 
diagnosticity  of  information  processing. 

5.  A  closed-loop  system  represents  a  method  for  the  use  of  psychophysiological  measures  in 
adaptive  automation  technology. 

6.  A  disadvantage  of  physiologically-based  adaptation  is  the  efficacy  of  different  adaptive 
algorithms  and/or  adaptive  logic  (similar  to  behaviour-based). 

7.  The  technology  for  physiologically-based  adaptation  is  not  yet  mature  enough. 

8.  Presently,  there  is  not  enough  existing  psychophysiological  research  to  provide  adequate 
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information  on  which  to  base  the  decision  for  adaptation. 

9.  It  is  suggested  that  psychophysiological  measures  could  be  used  not  only  as  an  input  signal 
for  the  regulation  of  automation,  but  also  to  assess  underlying  changes  accompanying 
performance  changes  during  development  of  adaptive  automation  systems. 


6.5  Combination-based  (Convergent)  Monitoring 

The  previous  sections  indicate  that  operator  state  monitoring  systems  are  able  to  capture  and 
process  a  large  amount  of  data.  Although  the  various  forms  of  data  can  be  treated  as  separate 
variables,  the  relationships  between  different  data  sources  also  contain  valuable  information. 
For  example,  the  absence  of  an  arousal  reaction  to  a  mild  threat,  such  as  a  low  altitude 
warning,  may  indicate  that  the  operator  is  confident  and  in  control.  However,  it  may  also 
indicate  a  loss  of  SA  caused  by  dangerously  low  levels  of  arousal.  In  recognition  of  the 
importance  of  convergent  processing,  combination-based  adaptation  systems  are  capable  of 
performing  complex  multivariate  analyses  in  order  to  improve  inferences  about  operator  state. 
A  further  benefit  of  convergent  processing  is  that  hidden  predictive  trends  can  often  be 
discovered  in  the  relations  between  data  sets  that  cannot  be  obtained  from  either  data  set 
alone.  Artificial  neural  networks  can  also  be  used  in  order  to  search  for  hidden  patterns  within 
data  (Pleydell-Pearce,  1999). 

A  characteristic  feature  of  human  behaviour  is  that  there  are  widespread  differences  in 
behavioural  and  physiological  responses  to  similar  situations.  This  means  that  conclusions 
that  are  based  upon  average  findings  from  a  group  of  individuals  may  correlate  only  weakly 
with  the  behaviour  of  a  particular  person.  However,  scientific  approaches  to  problems  such  as 
mental  workload  are  nearly  always  based  upon  data  averaged  across  subjects.  In  contrast,  very 
little  research  has  attempted  to  identify  unique  but  reproducible  changes  within  single 
individuals.  A  major  and  novel  feature  of  contemporary  combination-based  monitoring 
approaches  (e.g.,  the  Cognition  Monitor  of  the  Cognitive  Cockpit)  is  that  the  monitoring 
system  can  learn  about  the  behaviour  of  individuals,  and  look  for  predictable  regularities  in 
their  response  to  changing  patterns  of  workload  (Pleydell-Pearce,  1999). 


Reference:  t<Fh  I 

Duric,  Z.,  Gray,  W.D.,  Heishman,  R.,  Li,  F.,  Rosenfeld,  A.,  Schoelles,  M.J.,  Schunn.  C.,  and 
Wechsler,  H.  (2002).  Integrating  perceptual  and  cognitive  modeling  for  adaptive  and  intelligent 
human-computer  interaction.  In  Proceedings  of  the  IEEE,  90(7),  1272-1289. 


Overview: 

The  purpose  of  this  paper  was  to  describe  technology  and  tools  based  on  human  cognitive, 
perceptual,  motor,  and  affective  factors,  which  are  modeled  for  intelligent  systems.  The  authors 
propose  an  integrated  system  approach  which  measures  perceptual  and  cognitive  operator  states. 

Emotional  intelligence.  Recognition  of  affective  states  focuses  on  their  physical  form  (e.g.,  blinking 
or  face  distortions  underlying  human  emotions),  rather  than  implicit  behavior  and  function. 
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Intelligent  nonverbal  agent.  A  user  model  is  based  on  nonverbal  information,  such  as  facial 
expressions,  posture,  point  of  gaze,  and  the  speed  or  force  with  which  a  mouse  is  moved. 

Proposed  framework.:  The  methodology  integrates  nonverbal  information  with  a  computational 
cognitive  model  of  the  operator.  In  turn,  this  information  is  fed  to  the  system  so  that  it  adapts  the 
interface. 

The  are  four  main  modules  in  this  framework: 

•  Perceptual  module.  This  module  processes  images  of  the  face,  the  eye  (gaze  location  and 
pupil  size),  and  the  upper  body,  and  analyzes  their  relative  motions;  the  behavioral  module 
processes  information  about  actions  applied  to  the  computer  interface  directly,  such  as 
keystroke  choices,  the  strength  of  keystrokes,  and  mouse  gestures. 

•  Behavioral  Processing  module.  Processes  keystroke  (choice  and  rate)  and  mouse  data  (clicks 
and  movements). 

•  Cognition  module :  Reasons  possible  cognitive  states.  This  model  is  synchronized  with 
behavioral  and  perceptual  data.  All  three  modules  interact  with  each  other. 

•  Interface  module.  The  OMI. 

ACT-R  is  a  hybrid  architecture  to  interpret  the  information  from  the  four  modules  and  adapt 
accordingly.  ACT-R/PM  predicts  reaction  times  and  contains  probabilities  of  responses  of  motor 
movements,  shifts  of  visual  attention,  and  capabilities  of  human  vision.  The  cognition  module 
builds  a  detailed  mapping  of  the  interpretations  of  the  sensory-motor  data  onto  the  ACT-R/PM 
model. 


Conclusions  for  IASs: 

1 .  Perceptual  processing  can  include  lower  arm  movements,  facial  data  processing,  eye-gaze 
tracking,  and  mouse  gestures.  Additional  tools  are  possible,  including  upper  body  posture 
(head  and  shoulders).  Refer  to  the  article  for  further  details  on  the  tools  for  perceptual 
processing. 

2.  The  cognitive  and  emotional  states  of  a  person  can  be  correlated  with  visual  features  derived 
from  images  of  the  mouth  and  eye  regions. 

3.  Eye  movements  can  indicate  the  information  an  operator  is  attending  to.  For  instance,  eye 
tracking  is  a  popular  online  measure  of  high-level  cognitive  processing. 

4.  A  cognitive  model  has  the  ability  to  perceive  and  interact  with  the  external  world,  as  the 
operator  does. 

5.  ACT-R/PM  can  model  the  effects  of  fatigue  and  distraction  on  memory,  vision,  and  motor 
behavior  and  therefore  on  performance. 

6.  Models  of  cognition  could  become  an  important  tool  for  designers  of  real-time  safety-critical 
systems. 


Wilson,  G.F.  and  Russell,  C.A.  (2006).  Psychophysiologically  versus  task  determined  adaptive 
aiding  accomplishment.  Proceedings  of  the  2nd  Augmented  Cognition  conference,  San  Francisco, 
CA. 
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Overview: 


A  study  was  conducted  to  Investigate  two  methods  of  providing  adaptive  aiding  using 
psychophysiological  measures  to  assess  OFS  in  an  uninhabited  aerial  vehicle  (UAV)  task. 

Methodology 

Ten  subjects  performed  the  following  tasks:  monitoring  four  UAVs,  downloading  radar  images  of 
target  areas,  designating  targets,  and  giving  command  to  release  weapons.  Two  levels  of  task 
demand  were  used. 

Each  subject's  physiology  (five  EEG  channels,  EOG  and  ECG)  was  monitored.  An  artificial  neural 
network  was  used  to  determine  when  they  were  cognitively  overloaded. 

Adaptive  aiding  was  provided:  (1)  only  when  the  psychophysiologically  determined  OFS  indicated 
high  mental  workload  or;  (2)  initiated  at  the  first  instance  of  high  mental  workload  that  remained  on 
until  that  task  segment  ended.  The  first  provided  aiding  that  was  turned  on  and  off  several  times  as 
the  OFS  varied  while  in  the  second  the  aiding  remained  on  until  the  task  was  completed  regardless 
of  the  changes  in  physiology. 

Results:  Both  procedures  produced  statistically  significant  improvement  in  performance  compared 
to  a  “no  aiding”  condition.  The  psychophysiologically  determined  aiding  procedure  was  associated 
with  better  performance  than  the  second  method  but  was  not  significantly  better. 


Conclusions  for  IASs: 

1 .  Since  psychophysiological  measures  may  be  subject  to  fairly  rapid  fluctuations,  the  adaptation 
could  be  initiated  too  rapidly  which  could  interfere  with  an  operator’s  performance. 

2.  Real-time  analysis  of  EEG  has  been  used  to  modify  task  characteristics  to  better  match 
operator  functional  state. 

3.  In  tasks  where  a  wide  range  of  adaptations  are  available,  it  will  be  critical  to  match  the 
adaptation  with  the  specific  cognitive  resource. 


Reference: 


Wilson,  G.F.,  Russell,  C.A.,  and  Davis,  I.  (2006).  The  importance  of  determining  individual  operator 
capabilities  when  applying  adaptive  aiding.  Proceedings  of  the  50th  Human  Factors  and 
Ergonomics  Society  Conference,  San  Francisco,  CA. 


Overview: 


This  paper  outlines  a  study  to  assess  the  effect  of  adaptive  aiding  according  to  an  operator’s  skill 
level  at  varying  levels  of  task  difficulty.  It  was  found  that  the  best  performance  occurred  when 
adaptive  aiding  was  based  on  psychophysiological  data.  An  artificial  neural  network  was  used  to 
implement  the  adaptation. 


Conclusions  for  IASs: 

1 .  The  authors  recommend  implementing  adaptation  based  on  an  operator’s  cognitive  abilities 
and  skill  levels.  The  thresholds  for  initiating  adaptation  should  be  based  on  the  cognitive 
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capabilities  of  the  operator. 

2.  Aiding  is  most  effectively  provided  based  upon  the  psychophysiologically  determined  OFS. 


Sheridan,  T.B.,  and  Parasuraman,  R.  (2006).  Human-automation  interaction.  In  R.S. Nickerson 
(Ed.).  Reviews  of  Human  Factors  and  Ergonomics,  Volume  1.  HFES:  Santa  Monica,  CA. 


Overview: 

This  paper  reviews  recent  research  in  the  area  of  human-automation  interaction.  It  describes 
taxonomies  including  supervisory  control  of  automation  and  function  allocation,  and  models  of 
human-automation  interaction.  The  paper  outlines  automation-related  accidents  associated  with 
inadequate  feedback  and  misuse  of  automation,  and  evaluates  the  social,  political,  and  ethical 
issues  related  to  role  of  etiquette  and  trust  on  operator  performance. 

Delegation  interfaces:  In  these  systems,  the  operator  delegates  tasks  to  the  system,  at  times  of  the 
operator’s  own  choosing  and  receives  feedback  on  their  performance. 


Conclusions  for  IASs: 

1 .  Critical  events  method.  Automation  is  triggered  by  critical  events  (e.g.,  when  pilot  looses 
consciousness  the  auto-pilot  is  automatically  executed).  Critical  events  method  is  flexible  as  it 
can  be  coupled  with  mission  planning  but,  it  does  not  take  into  account  operator  requirements. 

2.  Operator  performance  and  physiological  measurements.  Automation  is  adapted  based  on  an 
assessment  of  operator  state.  For  instance,  this  could  include  using  a  secondary-task 
measurement  technique,  to  assess  operator  workload  when  performing  a  primary  task,  or  by 
using  EEG  and  ERP  measurement.  Measurement  of  operator  performance  or  physiological 
state  can  be  potentially  responsive  to  unpredictable  changes  in  operator  cognitive  states. 
Physiological  measures  can  be  designed  to  be  relatively  unobstrusive,  and  have  high 
bandwidth  compared  with  performance  measures.  A  disadvantage  is  measurement  sensitivity, 
which  needs  to  be  established  in  each  application  domain  and  is  only  as  good  as  the 
technology  itself. 

3.  Modeling:  A  set  of  pre-defined  rules  for  implementing  adaptive  automation.  Modelling 
techniques  can  be  implemented  offline  and  easily  incorporated  into  a  rule-based  expert 
system.  However,  a  valid  model  is  required  and  different  models  within  the  same  system  might 
give  contrary  decisions  at  particular  moments. 

4.  Hybrid:  A  mix  of  the  other  three  methods.  Hybrid  methods  attempt  to  optimize  relative  benefits 
and  disadvantages  of  each  technique  and  may  therefore  offer  the  best  general  approach  to 
implementing  adaptive  automation. 


6.5.1  Cognition  Monitor  (Cognitive  Cockpit,  United  Kingdom) 

The  Cognition  Monitor  module  of  the  United  Kingdom’s  Cognitive  Cockpit  programme 
provides  a  good  example  of  combination-based  adaptation  (i.e.,  using  a  combination  of 
behavioural  and  physiological  measures  to  infer  operator  state)  (Figure  1 1). 
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The  COGMON  has  32  analogue-to-digital  converters  capable  of  recording  physiological, 
behavioural,  and  situational  data  in  real  time.  These  permit  analysis  of  both  low  and  high 
frequency  electroencephalographic  measures.  The  system  can  perform  near  real  time  analysis 
of  all  32  channels  and  can  examine  coherence  functions  across  10  different  frequency 
bandwidths  for  each  channel.  These  data  provide  information  about  alertness,  but  also  allow 
inferences  to  be  made  about  the  nature  of  current  cognitive  activity. 


Figure  11:  Overview  of  Cognition  Monitor  Inputs  and  Outputs  (Willis,  2005). 

Blink  rate  and  eye-movement  activities  (i.e.,  saccades)  are  also  examined  in  detail.  These 
provide  information  about  visual  workload  and  can  be  used  to  determine  the  current  locus  of 
fixation. 

Heart  rate  and  heart  rate  variability  are  determined  within  COGMON.  Respiration  rate  is  also 
measured  and  analysed.  Muscle  tonus,  a  useful  measure  of  alertness  and  limb  activity,  is 
recorded  and  analysed  within  COGMON  via  electromyographic  biosensors.  Central  and 
peripheral  cutaneous  temperatures  are  also  measured  continuously  and  electrodermal  activity 
is  monitored.  These  measures  provide  information  about  activity  levels  in  the  autonomic 
nervous  system  (e.g.,  stress). 

Auditory  inputs  to  the  operator  via  headphones  are  logged  by  COGMON,  as  well  as  general 
environmental  noise.  Operator  vocalisations  are  also  monitored  using  microphones  and  via 
larynx  EMG.  Ambient  factors  including  luminance,  temperature  and  noise  are  also  monitored 
by  COGMON. 

The  COGMON  contains  a  large  amount  of  statistical  and  analytical  software.  These 
algorithms  are  aimed  at  performing  analysis  of  COGMON’ s  inputs.  They  are  also  designed  to 
process  data  from  different  sources  convergently.  Such  convergent  analyses  allow  a 
considerable  improvement  in  inferences  about  operator  state  compared  to  those  reliant  upon 
any  one  source  alone.  The  analytical  routines  are  also  aimed  at  identifying  pilot  ‘bespoke 
profiles’.  Indeed,  the  philosophy  underlying  COGMON  assumes  that  the  suitability  of 
workload  measures  may  vary  from  context  to  context,  and  from  pilot  to  pilot. 
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6.6  Summary 

Although  the  COGMON  does  not  claim  to  be  the  definitive  workload  assessment  system,  it 
does  represent  one  of  the  most  comprehensive  attempts  to  produce  such  a  system.  Common  to 
all  such  systems,  the  COGMON  is  constrained  by  the  maturity  and  availability  of  additional 
hardware  and  software.  For  example,  reliable  optical  tracking  of  visual  fixation  can 
considerably  enhance  the  monitor’s  ability  to  assess  visual  workload  and  determine  the 
current  locus  of  visual  attention.  In  addition,  its  development  is  in  part,  dependent  upon  the 
rate  of  progress  made  in  the  other  components  of  the  intelligent  adaptive  system.  For  example, 
software  aimed  at  monitoring  operator  interactions  with  system  controls  cannot  be  fully 
implemented  until  the  simulation  environment  is  reasonably  mature.  Similarly,  software 
aimed  at  the  receipt  and  analysis  of  general  situational  data  requires  that  the  situation 
assessment  sub-system  has  reached  a  sufficient  level  of  maturity.  Finally,  the  complexity  of 
systems  such  as  the  COGMON  means  that  its  inner  functions  may  be  quite  hard  to  visualise 
making  the  examination  of  COGMON’ s  current  internal  status  by  the  operator  extremely 
difficult.  The  Cognitive  Cockpit  programme  has  explored  the  use  of  HTML  format  (i.e., 
Internet  page  based)  to  represent  COGMON  states  (Figure  12). 


Figure  12:  Screenshot  of  Cognition  Monitor  visualisation  of  its  internal  workings.  41 
Inputs  are  presented  on  the  left,  and  18  outputs  (i.e.,  assessment  of  operator  load)  are 
presented  on  the  right  (from  Willis,  2005). 


Table  18  summarises  the  advantages  and  disadvantages  of  the  frameworks,  and  the 
relationship  between  the  frameworks  in  terms  of  authority,  agency  and  user  model,  and  which 
frameworks  are  applicable  to  a  given  situation  (i.e.,  domain  applicability). 
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Table  18:  Summary  of  adaptation  methods, 


Psychophysiological-based 

Adaptation 

Behaviour-based  Adaptation 

Combination-based  Adaptation 

Advantages 

Can  understand  how  the  brain  carries  out 
the  complex  tasks  of  everyday  life — and  not 
just  the  simple,  artificial  tasks  of  the 
research  laboratory. 

Measures  of  brain  function  can  indicate  not 
only  when  an  operator  is  overloaded, 
drowsy,  or  fatigued,  but  also  which  brain 
networks  and  circuits  may  be  affected. 

Analysis  of  learning  a  complex  task  can  be 
done  by  understanding  the  brain  changes 
that  accompany  stages  of  learning  could 
lead  to  the  development  of  better  training 
procedures. 

The  operator’s  psychophysiology  and  their 
performance  can  be  monitored  continuously 
with  little  intrusion. 

Can  detect  changes  in  cognitive  states  in 
real-time. 

Physiological  measures  can  be  designed  to 
be  relatively  unobtrusive,  and  have  high 
bandwidth  compared  with  performance 
measures. 

Implicit  behavioural  measurements  are  non- 
intrusive. 

Plan  generation  and  recognition  and  intent 
estimation  (e.g.,  operator  intent)  can  be  used 
to  adapt  the  system  based  on  operator 
behaviour  in  combination  with  a  goal/task 
model.  Algorithms  based  on  a-priori 
probabilities  for  possible  hypotheses  have 
proven  useful  for  recognizing  and  estimating 
operator  intent.  The  probabilities  can  be 
modified  with  respect  to  operator  actions. 

When  behavioural  measures  are  coupled 
with  psychophysiological  measures, 
changes  in  performance  may  be  more 
precisely  localized  to  central  processing 
stages  than  to  response-related  processing. 

Measures  of  brain  function  may  provide 
more  information  when  coupled  with 
behavioural  measures  than  behavioural 
measures  alone. 

Measurement  of  operator  performance  and 
physiological  state  has  the  advantage  of 
being  potentially  responsive  to 
unpredictable  changes  in  human  operator 
cognitive  states. 

Disadvantages 

Technology  is  still  immature  to  make 
psychophysiological  measures  applicable 
outside  the  laboratory. 

The  classification  of  operator  functional  state 
must  be  highly  accurate  in  order  to  provide 
useful  information  and  also  to  gain  operator 
acceptance. 

Higher  potential  for  noise  interference. 

The  “invisibility”  of  behaviourally-based 
measures  can  increase  OOTL  performance 
problems  and  decreases  situational 
awareness. 

Is  dependent  on  the  operator  model  (i.e. , 
task  based) 

Is  dependent  on  the  user  model  (i.e.,  task 
based). 

Need  to  ensure  that  both  types  of  measures 
are  match  and  correlate. 
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Psychophysiological-based 

Adaptation 

Behaviour-based  Adaptation 

Combination-based  Adaptation 

Psychophysiological  measures  may  be 
subject  to  fairly  rapid  fluctuations. 

Measurement  sensitivity  is  only  as  good  as 
the  technology  itself. 

Also  dependent  on  the  user  model 
(psychophysiological). 

Relationship  to 
Frameworks 

Personal  Web  Server;  Stock  Trader;  DRDC 
UAV  Project;  DIAManD;  Adaptive  Icon; 
ConCall;  Intelligent  Classroom;  CAMMA 

COGPIT,  PA/RPA 
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7  Guidance  for  the  Design  of  Intelligent  Adaptive 
Systems 


7.1  Introduction 

The  information  gathered  during  the  literature  review  activities  about  theoretical  frameworks, 
analytical  approaches,  multi-agent  systems,  and  the  use  of  psychophysiological-  and 
behaviour-based  feedback  systems  provide  guidance  for  the  design  and  development  of  an 
IAS.  This  report  has  highlighted  the  strengths  and  weaknesses  of  several  design  and  analysis 
approaches.  The  objective  of  this  section  is  to  outline  and  describe  a  development  route-map 
for  the  successful  development  of  an  IAS;  the  development  process  is  outlined  in  Figure  14. 
The  first  step  is  to  conduct  a  taxonomic  analysis  of  the  proposed  system,  followed  by  the 
selection  of  the  appropriate  framework,  analysis  methodology,  and  design  methodology.  The 
final  step  is  the  selection  of  the  appropriate  design  guidelines  (in  terms  of  principles  of 
adaptation,  interaction,  etc.).  Sections  12.2  through  12.7  describe  each  step  in  more  detail, 
with  particular  emphasis  on  the  selection  of  the  most  appropriate  method  for  the  development 
of  a  specific  IAS. 


Figure  13:  Development  route-map  for  Intelligent  Adaptive  Systems. 
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7.2  Conduct  Taxonomic  Analysis  of  Proposed  Intelligent 
Adaptive  System 

The  development  and  implementation  of  intelligent  adaptive  systems  can  be  guided  by  a 
taxonomic  approach  that  scopes  the  options  available  for  the  capability  and  functionality  of 
the  system.  In  addition,  a  taxonomic  approach  can  assist  the  creation  of  an  audit  trail  for  the 
design  of  the  system.  Finally,  it  provides  a  road-map  for  development  in  that  it  allows  the 
development  team  to  focus  on  specific  implementations  after  scoping  all  of  the  possibilities. 
Figure  14  describes  a  taxonomic  analysis  of  an  intelligent  adaptive  system  for  military  fast-jet 
ground  attack  operations.  There  are  three  axes:  what  mission-related  cockpit  tasks  are 
appropriate  for  machine  assistance,  the  degree  of  such  assistance,  and  the  cockpit  interfaces 
through  which  this  interaction  is  likely  to  occur. 


Mission  Timeline 


Figure  14:  Example  of  taxonomic  analysis  of  an  intelligent  adaptive  system  (from 

Banbury,  1999). 


In  order  to  create  such  a  taxonomy,  the  following  factors  need  to  be  defined: 

•  The  role  of  the  human  operator; 

•  The  role  of  the  decision  aid; 

•  The  level  of  automation  possible; 

•  The  number  of  behavioural  and  cognitive  functions  possible; 

•  The  operational  requirements  of  the  scenario  in  which  both  the  human  and  decision 
aid  were  expected  to  operate  in;  and, 

•  The  cockpit  interface  technologies  through  which  this  interaction  can  occur. 
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In  defining  these  factors,  the  approach  allows  responsibilities  to  be  allocated  between  the 
human  and  automated  system,  for  a  given  mission  segment,  and  through  a  specific  interface 
technology.  The  approach  also  allows  the  development  team  to  construct  an  appropriate 
mission  scenario  which  encompasses  the  range  of  system  functionality  and  capability 
identified  by  the  taxomony.  The  mission  scenario  is  used  in  both  the  subsequent  analysis  (i.e., 
as  a  precursor  to  the  functional  decomposition  of  tasks,  goals  and/or  functions)  and 
verification  (i.e.,  determination  of  measures  of  effectiveness  and  performance)  activities. 
Finally,  the  taxomic  approach  allows  the  development  team  to  quickly  scope  the  functionality 
and  capability  of  the  IAS  in  terms  priority  and  feasibility.  This  allows  the  development  team 
to  maximise  the  impact  of  the  IAS  on  operational  performance  whilst  reducing  development 
risk  (e.g.,  due  to  dependence  on  immature  technology)  within  the  time  and  budgetary 
constraints  of  the  project. 

7.3  Select  Development  Framework 

The  selection  of  an  appropriate  development  framework  affords  the  development  team  a 
number  of  advantages:  reduction  in  development  time  and  costs  from  leveraging  previous 
research;  benefit  from  the  lessons  learnt  from  past  projects;  and  providing  an  insight  into  the 
potential  operational  impact  of  the  developed  system.  One  of  the  most  recent  and 
comprehensive  attempts  to  generate  a  design  and  development  framework  for  IASs  was  by 
Edwards  (2004,  see  9.3.3  of  the  report).  Edwards  examined  a  variety  of  theoretical  approaches 
to  generate  a  generic,  integrated  and  comprehensive  framework  for  the  development  of  an 
intelligent,  adaptive,  agent-based  system  for  UAV/UCAV  control.  The  generic  framework 
comprised  the  following  design  approaches: 

•  CommonKADS  (and  MAS-CommonKADS) .  A  knowledge  management  and 
engineering  methodology  that  has  been  used  in  the  development  of  knowledge-based 
systems  (Schreiber,  Akkermans,  Anjewierden,  de  Hoog,  Shadbolt,  Van  de  Velde  and 
Wielinga,  2000); 

•  IDEF  Standards.  A  set  of  guidelines  similar  to  CommonKADS,  except  that  the 
guidelines  support  temporal  modeling  and  ontology  construction  more  effectively. 
IDEF  (ICAM  Definition)  was  developed  as  a  product  from  the  US  Air  Force 
Integrated  Computer  Aided  Manufacturing  (ICAM)  project; 

•  Explicit  Models  Design.  The  methodology  guides  the  construction  of  models  that 
identify  and  compartmentalise  the  knowledge  required  by  knowledge-based  systems. 
Explicit  Models  Design  has  the  potential  to  influence  how  the  intelligent  adaptive 
system  functions; 

•  Perceptual  Control  Theory.  Developed  by  Powers  (1990  a  &b),  and  subsequently  by 
Hendy,  Beevis,  Lichacz  and  Edwards  (2002),  the  theory  provides  a  feedback  control 
system  for  the  goal-directed  behaviour  of  the  knowledge-based  system.  Similar  to 
Explicit  Models  Design,  Perceptual  Control  Theory  has  the  potential  to  influence 
system  functioning;  and, 

•  Ecological  Interface  Design.  Developed  by  Vicente  and  Rasmussen  (1992), 
Ecological  Interface  Design  provides  a  set  of  techniques  for  the  design  of  the  OMI 
based  on  levels  of  cognitive  control  (i.e.,  skill,  rule  and  knowledge-based  control). 
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Edwards  argues  that  the  combination  of  these  multi-disciplinary  approaches  provides  a 
comprehensive  and  efficient  means  of  developing  intelligent  adaptive  systems.  The  output  of 
these  processes  is  the  construction  and  specification  of  a  number  of  models  that  are  used  to 
construct  the  intelligent  adaptive  system: 

•  Organisation  Model.  Constructed  using  CommonKADS,  the  model  incorporates 
knowledge  relating  to  the  organisational  context  that  the  knowledge-based  system  is 
intended  to  operate  in  (e.g.,  command  and  control  structures,  ISTAR  etc.); 

•  Task  Model  Constructed  using  a  combination  of  CommonKADS  (for  high  level  tasks 
and  functions),  and  Ecological  Interface  Design  and  Explicit  Models  Design  (for 
greater  decomposition  of  tasks  and  functions);  the  model  incorporates  knowledge 
relating  to  the  tasks  and  functions  undertaken  by  all  agents,  including  the  operator; 

•  Agent  Model.  Constructed  using  CommonKADS,  the  model  incorporates  knowledge 
relating  to  the  participants  of  the  system  (i.e.,  computer  and  human  agents),  as  well  as 
their  roles  and  responsibilities; 

•  User  Model.  Developed  using  Explicit  Models  Design,  the  model  incorporates 
knowledge  of  a  human  operator’s  abilities,  needs  and  preferences; 

•  System  Model.  Specified  using  Explicit  Models  Design,  the  model  incorporates 
knowledge  of  the  system’s  abilities,  needs,  and  the  means  by  which  it  can  assist  the 
human  operator  (e.g.,  advice,  automation,  interface  adaptation); 

•  World  Model.  Specified  using  Explicit  Models  Design,  the  model  incorporates 
knowledge  of  the  external  world,  such  as  physical  (e.g.,  principles  of  flight  controls), 
psychological  (e.g.,  principles  of  human  behaviour  under  stress),  or  cultural  (e.g., 
rules  associated  with  tactics  adopted  by  hostile  forces); 

•  Dialogue/Communication  Model.  Specified  using  a  combination  of  CommonKADS 
and  Explicit  Models  Design,  the  model  incorporates  knowledge  of  the  manner  in 
which  communication  takes  place  between  the  human  operator  and  the  system,  and 
between  the  system  agents  themselves; 

•  Knowledge  Model.  Specified  using  CommonKADS,  the  model  incorporates  a  detailed 
record  of  the  knowledge  required  to  perform  the  tasks  that  the  system  will  be 
performing;  and, 

•  Design  Model.  Created  using  CommonKADS,  the  model  comprises  the  hardware  and 
software  requirements  related  to  the  construction  of  the  intelligent  adaptive  system. 
This  model  will  also  specify  the  means  by  which  operator  state  is  monitored. 

Figure  16  illustrates  the  sequential  process  by  which  the  models  described  above  are  created. 

It  also  indicates  where  there  are  duplications  of  the  models;  the  three  task  models  and  two 
communication  models  can  both  be  combined.  The  generic  framework  posited  by  Edwards 
shares  many  similarities  to  the  other  frameworks  described  in  Section  9.3.3,  most  notably  the 
Cognitive  Cockpit,  which  also  borrowed  heavily  from  both  the  CommonKADS  and 
Ecological  Interface  Design  approaches.  Common  to  all  approaches  reviewed  in  this 
document  are  following  system  functions: 

•  Tracking  of  operator  goals/plans/intent  (and  progress  towards  them); 

•  Monitoring  of  operator  state; 
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•  Monitoring  of  world  state; 

•  Knowledge  of  the  effects  of  system  advice,  automation  and/or  adaptation  on  operator 
and  world  state  (i.e.,  closed-loop  feedback);  and, 

•  Bespoke  OMI  to  handle  the  interaction/dialogue  between  the  operator  and  the  system 
agents  (e.g.,  tasking  interface  manager). 


CommonKADS  Ecological  Interface  Design  Explicit  Models  Design 


Construct 

Organisation  Model 

[ISTAR  /  C2  Structures] 

Adapt 

►  Task  Model 
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[Top  Level  Functions] 
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Construct 
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Figure  15:  Generic  development  framework  for  Intelligent  Adaptive  Systems  (Edwards, 

2004). 


Furthermore,  the  models  described  can  also  be  mapped  on  to  the  generic  conceptual 
architecture  previously  described  in  Section  8.5.1.  Figure  16  illustrates  the  mapping  of 
Edward’s  generic  development  framework  to  the  generic  conceptual  architecture:  the  User 
Model  enables  the  physiological  monitoring  of  the  operator;  the  Task,  System  and  World 
Models  enable  the  monitoring  of  mission  plan/goal  completion,  task/activities,  as  well  as 
entities  and  objects  in  the  external  environment;  the  Knowledge  Model  enables  the  system  to 
provide  advice  to  the  operator,  automate  tasks,  or  adapt  the  OMI;  and  the  Dialogue  Model 
enables  the  means  of  interaction  between  the  system  and  the  operator. 
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*  OMI  Design  Guidelines 

•  HCI  Principles 


SITUATION 

ASSESSMENT 

*  Mission  Plan/Goal  Monitoring 
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OPERATOR 
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Figure  16:  Mapping  of  Edward’s  framework  to  generic  conceptual  architecture. 


Figure  17  provides  advice  for  the  selection  of  specific  frameworks  that  may  be  used  to  guide 
the  development  process  for  a  particular  target  domain  (e.g.,  lessons  learnt,  appraisal  of 
technological  maturity).  Frameworks  are  divided  into  two  categories,  reflecting  the  two 
streams  of  development  by  the  HCI  and  HF  communities: 

•  Civilian.  Personal  computer-based  applications  and  web-based  applications;  and, 

•  Military.  Error-critical  applications  (e.g.,  for  the  control  of  UAVs  and  piloting  of 
combat  aircraft). 

Military -based  frameworks  tend  to  be  used  more  for  complex  and  error-critical  applications. 
In  this  case,  civilian  aviation  applications  are  also  included  (e.g.,  CASSY).  Finally,  both 
framework  categories  are  broken  down  further  into: 


•  Task  Execution.  Automation  technologies  that  perform  specific  tasks  for  the  operator 
(in  blue);  and, 

•  Information  Management.  Decision  support  and/or  adaptive  interface  technologies 
that  assist  the  operator  manage  information  (in  red). 
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Figure  17:  Decision  tree  for  selection  of  design  frameworks. 


7.4  Select  Analysis  Methodology 

Analysis  methodologies  provide  the  OMI  communication,  visual  display  and  control 
requirements  needed  for  the  design  of  the  IAS,  as  well  as  a  functional  decomposition  of  the 
tasks  within  the  domain  envisaged  for  it.  The  results  of  this  analysis  are  used  to  populate  the 
models  described  in  section  12.3.  Figure  18  associates  each  of  these  models  with  the  relevant 
tools/methods/techniques  previously  described  in  this  report. 

Specifically: 

•  Cognitive  Analysis  Methodologies.  Contribute  to  the  construction  of  the  Task,  Agent 
and  User  Models  (Section  9.2); 

•  Task  Analysis  Methodologies.  Contribute  to  the  construction  of  the  Task,  Agent  and 
System  and  World  Models  (Section  9.2); 
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•  Human-Machine  Function  Allocation  and  Agent-based  Design  Principles.  Contribute 
to  the  construction  of  the  Agent,  Dialogue  and  Communication  Models  (Section 
10.2.2); 

•  Human-Machine  Interaction  and  Organisation  Principles.  Contribute  to  the 
construction  of  the  Dialogue  and  Communication  Models  (Section  10.2.2); 

•  IDEF5  Guidelines.  Contribute  to  the  construction  of  the  ontology  and  knowledge 
base.  This  is  then  used  to  enumerate  the  knowledge  captured  by  the  analysis  process 
(Section  8.4.4); 

•  Domain  Feasibility,  Cost-Benefit  Analysis  and  Principles  for  Closed-Loop 
Implementation.  Contribute  to  the  construction  of  the  Design  Model,  which  includes 
the  means  by  which  operator  state  is  monitored  (Section  11);  and, 

•  Human  Factors  and  Human  Computer  Interaction  Principles.  Contribute  to  the 
construction  of  the  OMI  and  related  systems  (Section  10.2).  The  design  process  might 
also  include  principles  from  Ecological  Interface  Design  (Section  9.3.4). 


CommonKADS  Ecological  Interface  Design  Explicit  Models  Design 


Figure  18:  Analysis  techniques  mapped  on  to  Edward’s  generic  framework. 
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Figure  20  provides  advice  for  the  selection  of  analysis  methodologies  based  upon  project 
constraints  that  may  be  used  to  guide  the  development  process  (e.g.,  lessons  learnt,  validation 
studies).  All  constraints  should  be  considered  as  a  whole,  and  in  practice,  it  is  likely  that 
compromises  amongst  the  constraints  will  be  required.  Project  constraints  are  described  in 
terms  of: 

•  Budget.  The  conduct  of  the  analysis  is  cost  efficient  and  time  consuming; 

•  Tasks  for  Analysis.  Types  of  tasks  that  need  to  be  analysed.  Methodologies  are  sub¬ 
divided  into  those  that  are  best  to  suited  to  analyse  Simple ,  Complex  and  Cognitive 
(e.g.,  involving  processes  such  as  decision  making  and  problem  solving)  tasks;  and, 

•  Analysts.  Project  personnel  that  will  be  required  to  conduct  the  analysis. 
Methodologies  are  sub-divided  into  those  that  are  Simple  to  Learn  and  Apply  and 
those  that  are  Difficult  to  Learn  and  Apply. 

Figure  19  also  illustrates  that  the  analysis  methodologies  tend  to  fall  into  two  distinct 
categories;  those  that  can  be  used  for  complex  tasks,  but  are  time-consuming  and  require  well- 
trained  analysts  (in  red),  and  those  that  can  be  used  for  simple  tasks,  in  less  time  and  with 
lesser- trained  analysts  (in  blue). 


Figure  19:  Decision  tree  for  selection  of  analysis  techniques. 
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7.5  Select  Design  Methodology 

Figure  20  provides  advice  for  the  selection  of  design  methodologies  that  may  be  used  to  guide 
the  development  process  of  a  particular  target  system  (e.g.,  lessons  learnt,  validation  studies). 
All  requirements  should  be  considered  together.  Design  methodologies  are  described  in  terms 
of: 

•  Multi-Agent  Requirements.  Related  to  the  complexity  of  the  system,  this  is  the 
requirement  for  the  system  to  utilise  multiple  software  agents; 

•  Feedback  Requirements.  The  requirement  for  a  closed-loop  system  (i.e.,  the  ability  of 
the  system  to  resample  operator  and  world  state  following  the  implementation  of 
adaptation); 

•  System  Complexity.  The  degree  to  which  the  system  is  expected  to  assist  the  operator 
in  a  range  of  tasks; 

•  Cognitive  Requirements.  The  degree  to  which  the  system  is  expected  to  support  the 
operator  in  cognitive-related  tasks  (e.g.,  decision  making  and  problem  solving);  and, 

•  Safety/Reliability  Requirements.  The  requirement  that  the  system  is  expected  to 
support  the  operator  in  error/reliability-critical  tasks  (e.g.,  aviation,  military). 


Figure  20:  Decision  tree  for  selection  of  design  methodologies. 
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•  Expensive/Time  Consuming.  The  conduct  of  the  design  methodology  is  expensive  and 
time  consuming;  and, 

•  End-Operator  Engagement.  The  requirement  that  potential  end-operators  of  the 
system  are  expected  to  take  part  in  the  design  and  development  process. 

7.6  Select  Operator-State  Monitoring  Approach 

Figure  21  provides  advice  for  the  selection  of  an  operator-state  monitoring  approach  that  may 
also  be  used  to  guide  the  development  process  of  the  target  system  (e.g.,  lessons  learnt, 
appraisal  of  technological  maturity).  All  requirements  should  be  considered  together 
Operator-state  monitoring  approaches  are  described  in  terms  of: 

•  Handling  Unpredictable  Events.  The  requirement  for  the  system  to  assess  operator 
state  for  unpredictable  events  (i.e.,  outside  of ‘normal’  operations); 

•  Detect  Implicit  Behaviours.  The  requirement  for  the  system  to  monitor  operator  state 
despite  little  or  no  observable  actions  by  the  operator  (e.g.,  cognitive  activities); 

•  Real-time  Monitoring.  The  requirement  to  determine  operator  state  in  real-time  (e.g., 
aviation-related  applications); 

•  Non-intrusive  Monitoring.  The  requirement  that  the  operator  is  not  aware  of  or 
impeded  by  the  monitoring  system; 

•  Detect  Explicit  Behaviours.  The  requirement  for  the  system  to  monitor  operator  state 
through  observable  actions  by  the  operator  (e.g.,  using  vehicle  controls); 

•  Intent  Monitoring.  The  requirement  for  the  system  to  determine  the  operator’s  intent 
(e.g.,  goals  and  objectives); 

•  Precision.  The  requirement  for  the  system  to  determine  operator  state  with  fine¬ 
grained  precision,  as  opposed  to  determining  whether  the  operator  is  merely  over¬ 
loaded  or  under-loaded; 

•  Mature  Technology.  Related  to  the  expected  timing  of  the  field-ability  of  the  system, 
the  degree  to  which  the  technology  is  mature  (i.e.,  ready  to  be  fielded)  or  is  in 
development; 

•  User  Model.  The  requirement  for  the  system  to  have  a  highly  accurate,  or  even 
bespoke,  user  model  (e.g.,  model  of  human  cognition,  abilities  etc.);  and, 

•  Immune  to  Noise  Interference.  The  requirement  that  the  system  is  immune  to 
electrical-magnetic  noise  interference  (e.g.,  EEG  monitoring  is  very  susceptible  to 
this  kind  of  interference  and  as  a  result  makes  it  deployment  in  aircraft  very  difficult). 


245 


Figure  21:  Decision  tree  for  selection  of  operator-state  monitoring  approaches. 

7.7  Comply  with  Design  Guidelines 

The  final  step  for  developers  is  to  consider  and  address  a  number  of  design  guidelines  before 
an  IAS  can  be  effectively  deployed  into  service.  The  design  guidance  outlined  in  Section  1 1 
provides  guidance  relating  to  human-agent  teamwork  and  organisation,  as  well  as  general 
aspects  of  the  OMI  (the  means  by  which  the  operator  interacts  with  the  systems  agents).  As  an 
example,  the  Tasking  Interface  Manager  of  the  Cognitive  Cockpit  is  described. 

General  OMI  design  guidance  can  also  be  found  in  the  report  “Development  of  Decision  Aid 
Implementation  Guidance  for  the  INCOMMANDS  Human  Factors  Design  and  Evaluation 
Guide”  (Banbury  and  Gauthier,  2007).  The  aim  of  this  document  was  to  support  the  design 
and  development  of  OMI  concepts  developed  as  part  of  the  INCOMMANDS  TDP  by 
providing  a  structured  and  comprehensive  set  of  design  guidelines  which  address  electronic 
support  systems  concepts.  The  document  is  comprised  of  guidance  relating  to  OMI  design 
goals  as  well  as  specific  guidance  relating  to  specific  classes  of  electronic  support  systems, 
including:  Information  Management  Aids  (guidelines  to  optimise  and  organise  the 
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information  presented  for  efficient  information  acquisition  and  synthesis);  Decision  Making 
Aids  (guidelines  to  support  efficient  and  effective  decision  making);  and  Control  and  Action 
Aids  (guidelines  to  minimise  operator  out-of-the-loop  problems).  Finally,  the  report  provides 
guidelines  relating  to  Design  and  Evaluation,  and  Training  and  Implementation. 

7.8  Summary 

The  previous  sections  have  identified  a  number  of  criteria  that  can  be  used  to  determine  which 
of  the  analysis  and  design  frameworks  and  methodologies  should  be  used  for  the  design  and 
development  of  a  specific  intelligent  adaptive  system.  In  general,  most  of  the  frameworks  and 
methodologies  are  relatively  generic  (i.e.,  context  independent)  and  scalable  (i.e.,  adaptable 
for  a  particular  application).  In  addition,  approaches  can  be  combined  to  capitalise  on  their 
unique  strengths,  whilst  mitigating  their  weaknesses.  The  selection  of  these  frameworks  and 
methodologies  can  therefore  be  reasonably  flexible,  which  is  fortunate  given  that  there  are  a 
number  of  other  constraints  that  might  have  a  greater  impact  on  the  selection  of  approaches. 
These  constraints  include: 

1.  Project  Constraints.  Constraints  related  to  the  procurement  or  research  project,  including 
schedule,  in-service  timescales,  budget,  and  availability  of  analysts  (and  their  level  of 
expertise); 

2.  Target  Domain  Constraints.  Constraints  related  to  the  target  domain  of  the  to-be- 
developed  system,  such  as  complexity,  criticality,  uncertainty,  and  environmental 
constraints  (e.g.,  especially  relevant  to  the  choice  of  operator  state  monitoring  systems); 

3.  Operator  Constraints.  Constraints  related  to  the  operator,  including  consequences  of  error 
and  overload,  what  support  is  needed,  how  much  support  is  needed,  and  who  needs  to  be 
in  control  (e.g.,  especially  relevant  in  combat  domains);  and, 

4.  Task  Constraints.  Determination  of  the  task  conditions  under  which  a  particular  IAS  may 
be  beneficial,  what  tasks  are  suitable  for  support  by  an  IAS,  and  under  what  conditions  are 
these  technologies  appropriate  (e.g.,  suitable  for  goal  and/or  operator  state  monitoring, 
triggering  conditions).  Finally,  determine  those  tasks  where  IAS  support  will,  in  the 
opinion  of  the  operator,  be  most  beneficial,  and  thus  enhance  the  likelihood  of  operator 
acceptance. 

Section  12.8.1  illustrates  the  complete  process  using  a  worked  example  of  the  Intelligent 
Classroom  IAH  system. 

7.8.1  Worked  Example:  Intelligent  Classroom 

Taxonomic  Analysis :  The  role  of  the  machine  is  to  dynamically  assist  the  human  lecturer 
(operator)  in  a  classroom  environment  (controls  camera,  automatic  slide-switcher).  The  role 
of  the  human  operator  is  to  simply  present  a  lecture.  The  level  of  automation  is  determined  by 
watching  and  observing  the  operator’s  behaviours  and  recognizing,  projecting  and  executing 
some  plan  to  accomplish  an  operator  goal  (i.e.,  match  the  operator’s  actions  to  a  set  of  known 
plans  and  execute  that  plan  to  achieve  some  goal).  The  operator,  in  executing  part  of  the  plan, 
expects  the  machine  to  do  its  part  of  the  plan  and  sets  the  operational  requirements  of  the 
scenario  in  which  both  the  human  and  the  machine  were  expected  to  operate  in.  The 
interaction  between  the  operator  and  the  machine  does  not  occur  through  an  OMI,  but  rather 
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through  a  camera  that  monitors  and  keeps  track  of  the  various  activities  pursued  by  the  human 
operator  which  is  goal-based  and  driven  by  task  recognition. 

Select  Development  Framework :  The  Intelligent  Classroom  is  a  civilian  program  that  executes 
a  task  normally  performed  by  a  human  operator.  The  resulting  framework  for  the  Intelligent 
Classroom  requires  three  models  including  Task,  System  and  World  Models  that  will  enable 
the  monitoring  of  mission  plan/goal  completion,  task/activities,  as  well  as  entities  and  objects 
in  the  external  environment. 

Select  Analysis  Methodology :  Figure  20  outlines  a  selection  process  of  analysis  methodologies 
based  upon  the  project  constraints  that  can  guide  the  development  process  of  the  Intelligent 
Classroom.  In  the  case  of  the  Intelligent  Classroom,  the  tasks  are  relatively  complex  and 
require  some  cognitive  analysis.  Therefore,  goal-directed  and  applied  cognitive  task  analyses 
would  be  the  most  appropriate  analyses  to  use.  The  budget  and  the  analysts  assigned  to 
perform  the  analysis  will  determine  the  depth  and  breadth  of  the  approaches. 

Select  a  Design  Methodology :  Figure  21  guides  the  development  process  of  a  particular  target 
system  in  terms  of  multi-agent  requirements,  feedback  requirements,  system  complexity, 
cognitive  requirements,  and  safety/reliability  requirements.  The  Intelligent  Classroom  is  a 
relatively  complex  system  that  requires  reliable  multiple  agents  to  monitor,  recognize  and 
execute  some  plan  (action).  Feedback  and  cognitive  requirements  are  not  important  for  the 
development  of  this  system  since  it  is  obvious  when  an  action  has  been  performed  (i.e., 
change  presentation  slide).  Ecological  Interface  Design  and  Explicit  Model  Design  appear  to 
be  the  most  appropriate  design  methodologies  to  guide  the  development  process  of  the 
system. 

Monitor  Operator  State :  To  adapt  the  system  dynamically,  operator  actions  and  inputs,  in 
terms  of  gestures  and  voice  recognition  must  be  monitored.  In  this  case,  the  operator-state  that 
must  be  identified  is  the  operator’s  actions  in  the  context  of  what  the  system  believes  the 
operator  is  doing.  To  accomplish  this,  the  system  must  handle  unpredictable  events,  monitor 
the  operator  state  in  real-time,  be  non-intrusive,  detect  explicit  behaviours  and  monitor  the 
operator  intent.  Therefore,  using  a  behaviour-based  approach  to  monitor  operator  state  is 
considered  the  most  appropriate  approach. 
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8  Conclusions 


8.1  Summary 

Research  pertaining  to  IASs  has  demonstrated  that  when  automation  and  interface  adaptation 
that  are  implemented  dynamically  and  intelligently  (i.e.,  in  response  to  changing  task 
demands  placed  upon  the  operator),  can  permit  the  chief  benefits  of  automation  (e.g., 
workload  regulation)  to  be  realised  without  most  of  the  drawbacks  associated  with 
conventional  or  static  automation  (e.g.,  loss  of  Situation  Awareness).  One  of  the  chief 
assumptions  underlying  the  use  of  IASs  in  many  of  the  domains  covered  by  this  review  is  that 
an  operator  can  control  a  process  during  periods  of  moderate  workload  and  hand-off  control 
of  particular  tasks  when  workload  either  rises  above,  or  falls  below,  an  optimal  level.  Specific 
task  demands  can  be  selected  and  modified  to  ensure  that  the  most  critical  tasks  are  attended 
by  the  operator,  and  an  optimal  level  of  workload  is  maintained.  Intelligent  Adaptive  Systems 
are  also  sensitive  to  mission  context,  in  that  they  adapt  to  both  operator  and  mission/goal 
requirements. 

The  literature  research  obtained  during  this  project  achieved  the  following  goals: 

•  Identify  the  advantages,  disadvantages  and  applicability  of  development  frameworks, 
analysis  methodologies,  design  approaches,  and  operator-state  monitoring 
approaches; 

•  Make  some  progress  in  unifying  the  hitherto  independent  HF  and  HCI  approaches  to 
the  development  of  IASs  by  providing  a  generic  conceptual  framework  (i.e.,  R-A-A: 
role,  agency  and  authority)  and  a  generic  conceptual  architecture  which  map  to  both 
approaches  by  focusing  on  system  functionality  and  capability;  and, 

•  Develop  guidance  for  developers  to  assist  in  the  successful  design,  development  and 
implementation  of  IASs. 

8.2  The  Way  Ahead 

Several  technologies  were  once  major  stumbling  blocks  to  exploiting  IAS  concepts.  However, 
interest  in  these  technologies  external  from  the  military  domain  has  allowed  these 
technologies  to  reach  maturity.  These  technologies  can  be  summarised  as  follows  (Geddes 
and  Shalin,  1997): 

•  Increases  in  computing  power  and  associated  technologies; 

•  Development  of  object-oriented  computer  languages  that  are  especially  suited  to 
support  intelligent  aiding  systems  because  of  their  use  of  abstraction; 

•  Continued  progress  in  Artificial  Intelligence  has  provided  a  large  number  of  mature 
reasoning  algorithms  and  operational  knowledge  representations; 

•  Knowledge  capture  methods  are  well  developed  for  dealing  with  multiple  knowledge 
sources  and  tacit  knowledge;  and, 
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•  Architectures  for  large-scale  intelligent  systems  have  been  developed  that  support  a 
distributed,  co-operative  environment. 

The  evaluation  of  IASs  is  a  controversial  endeavour.  Much  of  the  past  empirical  research  on 
evaluating  such  systems  has  drawn  on  techniques  and  models  from  experimental  psychology. 
These  experimental  techniques  tend  to  reflect  the  evaluation  of  attributes  of  large  populations 
when  confronted  with  frequently  recurring  treatments.  However,  Geddes  and  Shalin  (1997) 
view  this  approach  as  inappropriate  for  IASs,  in  which  the  total  population  of  operators  is 
normally  small,  and  the  significant  stimuli  are  rare.  Geddes  and  Shalin  suggest  that  the 
mismatch  between  evaluation  methods  and  the  nature  of  IASs  has  resulted  in  many 
experimental  evaluations  failing  to  detect  any  significant  performance  differences  between 
aided  and  unaided  conditions.  Yet,  subjectively,  the  operators  report  strong  reactions  to  the 
aids,  both  favourable  and  unfavourable.  The  authors  argue  that  unless  evaluation  methods  are 
developed  and  accepted  by  the  development  community,  there  is  a  risk  that  progress  towards 
successful  IASs  will  be  constrained  by  an  apparent  lack  of  value. 

The  potential  benefits  to  successful  implementation  of  IAS  are  high;  potential  benefits  include 
a  reduction  in  accidents  and  incidents,  greater  economic  efficiency  and  reliability  of  systems 
and  operations,  and  greater  combat  effectiveness  of  military  operations.  However,  the 
potential  difficulties  of  applying  IASs  to  military  and  civilian  domains  are  also  significant. 

The  inclusion  of  new  IAS  technologies  does  not  reduce  the  need  for  Human  Factors  input  but 
simply  redirects  it.  Consideration  of  the  human  element  within  the  system  is  essential  to  the 
design  and  development  of  IASs,  in  order  to  ensure  that  incidents  and  accidents  are  mitigated, 
and  the  expected  benefits  to  safety  and  mission  effectiveness  are  realised. 

Finally,  despite  the  synergy  between  research  within  the  fields  of  Human  Factors  and  Human- 
Computer  Interaction,  there  is  a  pressing  practical  need  for  these  two  research  domains  to 
exploit  the  lessons  learnt  and  leverage  research  findings  from  each  other.  Efforts  towards 
these  goals  should  prove  extremely  fruitful. 
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9  Acronyms 


AA 

Adaptive  Automation 

ACTA 

Applied  Cognitive  Task  Analysis 

ACT-R 

Adaptive  Control  of  Thought-Rational 

ACT-R/PM 

Adaptive  Control  of  Thought- 
Rational/Perceptual  Motor 

ACWA 

Applied  Cognitive  Work  Analysis 

AI 

Artificial  Intelligence 

ANS 

Autonomic  Nervous  System 

ARP 

Applied  Research  Package 

ARP 

Applied  Research  Program 

AST 

Associate  System  Technology 

ATM 

Air  Traffic  Management 

AugCog 

Augmented  Cognition 

AUI 

Adaptive  User  Interface 

AWW 

Anti- Air  Warfare 

BAe 

British  Aerospace 

C3I 

Command,  Control,  Communication  and 
Intelligence 

CADM 

Core  Architecture  Data  Model 

CAMMA 

Crew  Assistant  Military  Aircraft 

CASSY 

Cockpit  Assistant  System 

CDAS 

Cognitive  Decision  Aiding  System 

CE 

Co-Pilote  Electronique 

CHI 

Computer-Human  Interaction 

CIE 

Crew  Intent  Estimation 

CM 

Cognition  Monitor 

CPU 

Central  Processing  Unit 

COGMON 

Cognitive  Monitor 

COGPIT 

Cognitive  Cogpit 

251 


ConTA 

Control  Task  Analysis 

CSE 

Cognitive  System  Engineering 

CTA 

Cognitive  Task  Analysis 

CWA 

Cognitive  Work  Analysis 

DAI 

Dynamically  Adaptive  Interface 

DAT 

Decision  Aiding  Taxonomy 

DBMS 

Data  Base  Management  System 

DERA 

Defence  Evaluation  and  Research  Agency 

DIAManD 

Decision-theoretic  InterAction  Manager  for 
Discourse 

DGMS 

Dialogue  Generation  and  Management  System 

DoDAF 

US  Department  of  Defence  Architectural 
Framework 

DRDC 

Defence  Research  and  Development  Canada 

DSS 

Decision  Support  Systems 

DTM 

Data  Transfer  Module 

ECG 

ElectroCardioGram 

EEG 

Electroencephalography 

EMD 

Explicit  Models  Design 

EMF 

Electro-Magnetic  Frequency 

EMG 

ElectroMyoGraphic 

EID 

Ecological  Interface  Design 

EOB 

Electronic  Order  of  Battle 

EOF 

Electro-OculoGraphic 

ERN 

Error-Related  Negativity 

ESS 

Electronic  Support  Systems 

ESSM 

Electronic  Sensor  Surveillance  Measure 

ERPs 

Event-Related  Potentials  (ERP)s 

FAN 

Functional  Abstraction  Network 

FCBA 

Future  Carrier-Borne  Aircraft 

fMRI 

functional  Magnetic  Resonance  Imaging 

FOAEW 

Future  Organic  Airborne  Early  Warning 
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FOAS 

Future  Offensive  Air  System 

FOAS 

Future  Offensive  Air  System 

GIM 

Global  Implicit  Measures  (of  Situational 
Awareness) 

GOMS 

Goals,  Operators,  Methods,  Selection  rules 

GSR 

Galvanic  Skin  Response 

GTA 

Goal  Directed  Task  Analysis 

GUI 

Graphical  User  Interface 

HCD 

Human-Centered  Design 

HCI 

Human-Computer  Interaction 

HEC 

Human  Electronic  Crewmember 

HMI 

Human-Machine  Interface 

HF 

Human  Factors 

HFM 

Human  Factors  and  Medicine 

HGA 

Hierarchical  goal  analysis  (HGA) 

HOTAS 

Hands  On  Throttle  And  Stick 

HTA 

Hierarchical  Task  Analysis 

IAA 

Intelligent  Adaptive  Automation 

IAH 

Intelligent  Adaptive  Hybrid 

IAI 

Intelligent  Adaptive  Interfaces 

IAS 

Intelligent  Adaptive  System 

ICF 

Intelligent  Control  Framework 

IOOD 

Intelligent  Object-Oriented  Design 

IP 

Information  Processing 

IPSS 

Integrated  Passive  Sensor  System 

ISU 

In-Service  Update 

JAD 

Joint  Application  Design  /  Development 

KA 

Knowledge  Acquisition 

KBS 

Knowledge  Based  System 

LAP 

Low  Altitude  flight  Planner 

LSPA 

Learning  System,  Pilot  Aiding 

MBMS 

Model  Based  Management  System 
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MEG 

MagnetoEncephaloGraphy 

MFTA 

Multi  Function  Task  Analysis 

MiDAS 

Mission  Displays  for  Autonomous  Systems 

MoD 

Ministry  of  Defence 

MOE 

Measures  of  Effectiveness 

MRI 

Magnetic  Resonance  Imaging 

OFS 

Operator  Functional  State 

OMI 

Operator  Machine  Interface 

OODA 

Observe  Orient  Decide  Act 

OV 

Operations  View 

OWL 

Web  Ontology  Language 

PA 

Pilot’s  Associate 

PACT 

Pilot  Authorizing  and  Control  Tasks 

PCT 

Perceptual  Control  Theory 

PET 

Positron  emission  tomography 

PGG 

Plan-Goal  Graph 

PIER 

The  Pilot  Intent  and  Error  Recognition 

PVI 

Pilot  Vehicle  Interface 

P-VACS 

Playbook-enhanced  Variable  Autonomy 

Control  System™. 

RAA 

Roles-Agent-  Authority 

RAF 

Royal  Air  Force 

RPA 

Rotorcraft  Pilot’s  Associate 

RPD 

Recognition  Primed  Decisions 

RTIC/RTOC 

Real  Time  Into  the  Cockpit  /  Real  Time  Out  of 
the  Cockpit 

SA 

Situation  Awareness 

SAM 

Surface-to-Air  Missile 

SASS 

Situation  Assessment  Support  System 

SATCOM 

Satellite  Communication 

SAWA 

Situational  Awareness  Assistant 

SM 

Sensor  Manager 
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SME 

Subject  Matter  Expert 

SOCA 

Social  Organization  and  Cooperation  Analysis 

sv 

Systems  View 

SWRL 

Semantic  Web  Rule  Language 

TCD 

Transcranial  Doppler  sonography 

TIBS 

Tactical  Information  Broadcast  Service 

TIM 

Tasking  Interface  Manager 

TMS 

Transcranial  Magnetic  Stimulation 

TSI 

Tactical  Situation  Interpreter 

TV 

Technical  Standards  View 

UAV 

Uninhabited  Aerial  Vehicle 

UCAVs 

Uninhabited  Combat  Air  Vehicles 

UCD 

User-Centered  Design 

US  AFRL 

United  States  Air  Force  Research  Laboratory 

USAF 

United  States  Air  Force 

WCSS 

Work-Centered  Decision  Support 

WDA 

Work  Domain  Analysis 

WSCE 

Weapon  System  Concept  Engineering 
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provide  guidelines  for  the  design  and  implementation  of  IASs.  Finally,  a  number  of  criteria 
that  can  be  used  to  select  appropriate  analytical  techniques  and  design  approaches  were 
also  developed.  The  proposed  frameworks  not  only  provide  guidelines  for  designing  IASs 
in  the  military  domain,  but  also  guide  the  design  of  other  generic  systems  to  optimize 
human-machine  system  performance. 

(U)  II  est  possible  d’ameliorer  considerablement  les  performances  des  ensembles 

homme-machine  en  ayant  recours  a  des  technologies  qui  peuvent  adapter  intelligemment 
I’interface  operateur-machine  (IOM)  et/ou  I’automatisation  des  taches  et  le  soutien 
accorde  a  I’operateur  conformement  au  contexte  externe  (c.-a-d.  le  contexte  de  la  tache) 
et  au  contexte  interne  (c.-a-d.  I’etat  de  I’operateur).  Toutefois,  I’absence  de  lignes 
directrices  etablies  en  matiere  de  conception  constitue  un  lourd  obstacle  a  la  conception 
efficiente  de  systemes  adaptatifs  intelligents  (SAI).  Un  examen  approfondi  de  la 
documentation  a  ete  effectue  afin  d’examiner  les  demarches  actuelles  en  conception  des 
SAI  et  un  cadre  de  travail  unifie  a  ete  elabore  afin  de  decrire  des  perspectives 
conceptuelles  en  faisant  appel  a  une  terminologie  uniforme  et  non  ambigiie.  Par  ailleurs, 
en  combinant  des  methodes  de  conception  des  domaines  des  interactions 
homme-ordinateur  (IHO)  et  des  facteurs  humains  (FH),  nous  avons  elabore  des  cadres 
conceptuels  et  de  design  afin  d’elaborer  des  lignes  d’orientation  afin  d’aider  a  la 
conception  et  a  la  mise  en  oeuvre  de  SAI.  Un  certain  nombre  de  criteres  de  selection  des 
methodes  analytiques  et  conceptuelles  appropriees  ont  aussi  ete  developpes.  Les  cadres 
recommandes  ne  guideront  pas  seulement  la  conception  des  SAI  dans  les  domaines 
militaires,  ils  aideront  aussi  dans  le  domaine  des  systemes  civils  afin  d’optimiser  les 
performances  des  systemes  homme-machine. 

1 4.  KEYWORDS,  DESCRIPTORS  or  IDENTIFIERS  (Technically  meaningful  terms  or  short  phrases  that  characterize  a 
document  and  could  be  helpful  in  cataloguing  the  document.  They  should  be  selected  so  that  no  security  classification  is  required. 

Identifiers,  such  as  equipment  model  designation,  trade  name,  military  project  code  name,  geographic  location  may  also  be  included.  If 
possible  keywords  should  be  selected  from  a  published  thesaurus,  e.g.  Thesaurus  of  Engineering  and  Scientific  Terms  (TEST)  and  that 
thesaurus  identified.  If  it  is  not  possible  to  select  indexing  terms  which  are  Unclassified,  the  classification  of  each  should  be  indicated  as 
with  the  title.) 

(U)  intelligent  adaptive  system;interface  design;human-machine 

system  ;human-machine  interaction  operator  machine  interface;human 
computer  interaction ;agent-based  system ;design  framework;analytical 
method ;behaviour-based  system;physiological-based  system;human 
automation  interaction ;adaptive  automation 


